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ABSTRACT 


Recognizing  the  need  to, enhance  flexibility  and  reduce 
the  burden  of  Government -funded  science  and  technology 
contracts.  Congress  crafted  Section  845  Other  Transaction 
Authority  (OTA)  to  release  Defense  Advanced  Research 
Projects  Agency  and  the  Services  from  complying  with 
statutes  and  regulations  in  the  FAR/DFARS  procurement 
process.  This  greater  flexibility  was  intended  to  attract 
commercial  firms  that  normally  would  not  do  business  with 
the  Government,  thus  expanding  the  defense  technology  and 
industrial  base.  This  study  was  conducted  to  identify  and 
develop  appraisal  metrics  that  could  be  used  to  measure 
both  the  use  and  value  of  Section  845  OTs .  The  thesis  also 
presents  a  survey  of  standard  contract  management  metrics 
used  by  various  buying  organizations.  The  researcher  found 
survey  respondents  rated  13  standard  contract  metrics  to  be 
appropriate  for  Section  845  OTs;  and,  recommends 
establishing  four  measures  to  serve  as  a  core  set  of 
metrics  applicable  to  all  Section  845  OTs. 
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I .  INTRODUCTION 

A.  GENERAL 

In  February  1996,  the  Director  of  Defense  Procurement 
(DDP) ,  Office  of  the  Under  Secretary  of  Defense  (OUSD) , 
Acquisition  and  Technology  (A&T)  briefed  the  Defense 
Manufacturing  Council  (DMC)  on  Defense  Advanced  Research 
Projects  Agency's  (DARPA)  use  of  Other  Transaction 
Authority  (OTA) .  [Ref.  l:p.  1]  The  briefing  reviewed  ten 
U.S.C  2371  instances  of  OTA  and  compared  and  contrasted 
OTAs  with  standard  contract  provisions .  Some  members  of 
the  DMC  questioned  why  the  Services  were  not  using  other 
transactions  (OTs)  to  conduct  research  and  development 
(R&D)  and  weapon  systems  prototype  projects  "better, 
faster,  and  cheaper."  As  a  result,  OUSD/DDP  established  an 
Integrated  Product  Team  (IPT)  to  examine  the  potential  to 
expand  OTs  in  the  Services. 

In  June  1996,  the  IPT  released  its  findings  in  a  Final 
Report.  This  report  cited  seven  primary  recommendations 
for  increasing  the  Services'  flexibility  and  encouraging 
them  to  use  OTs  for  research  and  development  projects. 
Recommendation  number  seven  suggested  the  following  action: 

Metrics  that  measure  the  use  and  value  of 
'flexible'  cooperative  agreements  and  Other 
Transactions  need  to  be  developed  and  employed. 

[e.g.,  this  could  be  as  straightforward  as  the 
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number  of  non-traditional  firms  attracted.]  [Ref. 
l:p.  3] 


The  report  cited  six  additional  recommendations  based  on 
the  assumption  that  the  Services  would  be  granted  Section 
845-type  authority  in  Fiscal  year  1997.  Section  845  of  the 
1994  Department  of  Defense  (DOD)  Authorization  Act 
broadened  the  use  of  OTs  by  giving  DARPA  the  authority  to 
carry  out  prototype  projects  directly  relevant  to  weapon 
systems.  Recommendation  number  six  stated  that  the 
following  action  should  be  taken: 


Activities  that  enter  into  Section  845 
transactions  should  develop  and  employ  metrics 
that  measure  the  use  and  value  of  Section  845 
'Other  Transactions'  for  prototype  acquisitions. 
These  metrics  should  provide  insight  into  whether 
the  845  'other  transactions'  are  'better,  faster, 
and  cheaper'  methods  of  conducting  business. 

[Ref.  l:p.  4] 


As  a  result  of  this  report,  OUSD  (A&T)  formed  several 
IPTs,  one  of  which  focused  on  developing  a  common  set  of 
performance  metrics  applicable  to  OTs.  Congress  and  OUSD 
(A&T)  both  considered  metrics  important  from  an  oversight 
and  internal  management  perspective.  The  intent  of  the 
Metrics  IPT  study  was  to  provide  OSD  and  the  Services  with 
a  means  of  assessing  the  value  of  this  authority;  what 
goals  did  the  Services  hope  to  accomplish  with  OTA,  and 
were  they  achieving  those  goals?  It  was  also  hoped  that 
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the  study  would  gather  data  upon  which  Congress  could  base 
a  decision  to  extend  OTA  under  Section  2371  of  Title  10, 
United  States  Code  (U.S.C.).  [Ref.  2]  Under  10  U.S.C.  2371, 
DOD  is  authorized  to  carry  out  research  projects  using 
"transactions  other  than  contracts,  cooperative  agreements, 
and  grants,"  also  known  as  OTs.  [Ref.  3]  The  authority  is 
due  to  expire  at  the  end  of  Fiscal  year  2001.  [Ref.  4] 

The  IPT's  preliminary  conclusion  was  that  the  Services 
were  achieving  some  financial  savings  and  shortening 
prototype  cycle  times.  However,  the  IPT  was  uncertain  how 
to  effectively  measure  the  benefits  of  OTs.  One  member  of. 
the  IPT  said: 

Team  members  need  to  better  understand  how  OT's 
are  being  used  before  we  reach  agreement  on  the 
usefulness  of  metrics .  Traditional  contractors 
are  doing  business  much  differently  these  days, 
making  access  to  sub-contractors  more  difficult. 

That's  where  the  really  technology  is  being 
exchanged.  [Ref.  5] 

For  the  period  of  fiscal  years  1990  to  1997,  the  DOD 
issued  210  OTs  and  Section  845  prototype  agreements  valued 
at  about  $3.4  billion.  [Ref.  6:p.  i]  In  August  1998,  the 
Office  of  the  Inspector  General,  DOD  (DODIG) ,  released 
Audit  Report  98-191  on  the  financial  and  cost  aspects  of 
OTs.  [Ref.  6]  The  report  reviewed  77  OTs  from  the  Army 
Communications  Electronics  Command  (CECOM) ,  Air  Force 
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Wright  Laboratory,  Defense  Contract  Management  Command 
centers  in  Seattle  and  Syracuse,  and  DARPA  valued  at  $1.7 
billion.  [Ref.  6:p.  ii]  The  report  concluded  that  the 
Services  needed  to  improve  their  management  controls  of 
OTs;  specifically,  management  had  not  devoted  sufficient 
attention  to  developing  metrics  to  quantify  the  benefits 
from  using  OTs.  The  report  recommended  that  the  Directors, 
Defense  Research  and  Engineering  and  Defense  Procurement, 
establish  quantifiable  performance  measures  to  determine 
the  costs  and  benefits  of  using  OTs.  [Ref.  6:p.  6] 

B.  OBJECTIVES  OF  THE  RESEARCH 

The  primary  objective  of  this  research  is  to  identify 
and/or  develop  appraisal  metrics  that  measure  both  the  use 
and  value  of  Section  845  "Other  Transactions."  This 
research  will  also  identify  the  broad  range  of  quantitative 
and  qualitative  appraisal  measurements  (metrics)  used  in 
contract  management . 

C.  RESEARCH  QUESTIONS 

1.  Primary  Research  Question 

To  what  extent  are  metrics  used  to  measure  standard 
contractual  transactions  appropriate  to  manage  Section  845 
"Other  Transactions?" 
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2.  Subsidiary  Research  Questions 

Additional  research  questions  are  as  follows: 

1.  What  are  Other  Transactions,  and  how  are  they 
distinguished  from  standard  contractual 
transactions  ? 

2 .  What  metrics  are  commonly  used  to  manage 
standard  contractual  transactions? 

3 .  To  what  extent  are  these  metrics  suitable  for 
managing  Other  Transactions  and  Section  845 
Prototype  Transactions? 

4 .  To  what  extent  can  new  metrics  be  developed  that 
are  appropriate  for  managing  Section  845  Other 
Transactions? 

5.  Can  a  core  set  of  metrics,  common  to  all  Section 

845  Other  Transactions,  be  developed? 

6 .  What  are  the  advantages  of  developing  metrics  for 
managing  Section  845  Other  Transactions,  and  what 
are  the  barriers  to  implementing  them? 

D.  SCOPE  AND  METHODOLOGY 

To  identify  appraisal  metrics  that  measure  both  the 
use  and  value  of  Section  845  OTs,  the  author  first  reviewed 
the  relevant  literature,  including  but  not  limited  to  the 
following : 

1.  References,  publications  and  electronic  media 
available  at  the  Naval  Postgraduate  School. 

2.  Published  academic  text  books  and  research  papers. 

3.  Internet  websites  and  homepages  (DOD  and  academic). 
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The  author  then  contacted  Acquisition  Professionals, 
Program  Managers,  and  Contract  Specialists  from  OUSD  (A&T) , 
Assistant  Secretary  of  the  Navy  (Research,  Development  & 
Acquisition) ,  Defense  Advanced  Research  Projects  Agency 
(DARPA) ,  Office  of  Naval  Research  (ONR) ,  Defense  Logistics 
Agency  (DLA) ,  Defense  Contract  Management  Command  (DCMC) , 
National  Aeronautics  and  Space  Administration  (NASA) , 
Department  of  Energy  (DOE)  and  major  DOD  buying  commands. 
The  research  involved  two  surveys  and  telephone  interviews 
with  at  least  one  representative  from  each  of  these 
organizations.  All  respondents  were  assured  of  anonymity. 
The  first  survey  was  conducted  to  obtain  current 
information  on  contract  management  metrics.  The  second 
survey  was  sent  to  senior-level  acquisition  professionals 
with  experience  in  drafting  and/or  administering  OTs.  This 
methodology  consisted  of  the  following  steps: 

1.  Conduct  a  thorough  review  to  identify  all  contract 
management  metrics  currently  being  used  by  DCMC, 
NASA  and  major  system  buying  commands. 

2.  Examine  contract  management  metrics  most  applicable 
to  R&D  and  prototype  contracts. 

3.  Prepare  a  survey  to  target  potential  OT  metrics. 

4.  Identify  a  common  set  of  credible  OT  metrics. 

5.  Evaluate  the  benefits  of  OT  metrics  found  by  using 
the  survey  and  interviews  with  senior-level 
acquisition  professionals. 
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SURVEY 

DCMC,  DLA,  DARPA,  AFRL, 
SPAWAR,  NAVSEA,  NAVICP,  NASA, 
DOE,  CAPS 


CH  IV 


TRADES 

Six  Categories 
of  Measurement 


E.  ORGANIZATION  OF  STUDY 

The  thesis  is  divided  into  six  chapters.  Chapter  I, 
the  introduction,  identifies  the  focus  and  purpose  of  the 
thesis'  and  states  the  primary  and  subsidiary  questions. 
Chapter  II  reviews  the  necessary  background  information  on 
the  legislative  and  operational  intent  of  Section  845, 
"Other  Transactions."  Chapter  III  summarizes  background 
information  concerning  metrics.  Chapter  IV  categorizes  and 
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analyzes  information  collected  in  the  first  survey. 

Chapter  V  analyzes  the  information  collected  in  the  second 
survey.  And  Chapter  VI  provides  the  study's  principal 
conclusions  and  recommendations.  The  final  chapter  also 
answers  the  research  questions  and  identifies  areas  for 
future  study. 

The  thesis  also  contains  four  appendices.  Appendix  A 
provides  a  survey  of  Contract  Management  Performance 
Metrics  currently  being  used  by  NASA,  DOE,  DCMC,  and  DOD 
buying  commands.  Appendix  B  provides  a  listing  of  contract 
management  metrics  by  classification  category.  Appendix  C 
provides  a  list  of  the  25  standard  contract  metrics 
subjectively  determined  to  be  most  relevant  to  measuring 
OTs.  And  Appendix  D  presents  a  list  of  terms  and 
definitions  applicable  to  "other  transactions." 


F.  BENEFITS  OF  THE  STUDY 

This  study  selects,  from  a  broad  range  of  contract 
management  metrics,  a  core  set  of  simple,  accurate  and 
credible  metrics  aimed  at  measuring  the  performance  of 
Section  845  OTs.  Such  metrics  would  strengthen 
Congressional,  OUSD  (A&T) ,  Defense  Contract  Audit 
Agency  (DCAA) ,  Services',  and  industries'  understanding  and 
belief  in  the  authority.  This  ultimately  serves  both 
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Government  and  industry  by  cooperatively  minimizing  the 
costs  of  research  and  development  projects.  By  surveying 
senior  level  acquisition  managers  who  have  used  Section  845 
OTA,  this  study  will  provide  information  beneficial  to  DOD 
buying  agencies  and  acquisition  professionals. 

Additionally,  this  study  will  conduct  a  survey  of 
contract  management  appraisal  and  monitoring  metrics 
currently  used  by  senior  managers  at  DLA,  DCMC,  NASA,  DOE, 
commercial  firms,  and  major  DOD  buying  commands.  This 
survey  could  yield  valuable  information  about  the 
acquisition  and  contracting  field  in  determining  where 
future  research  may  be  required. 
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IX.  OVERVIEW  OF  LEGISLATIVE  AND  OPERATIONAL  INTENT  OF 
SECTION  845  "OTHER  TRANSACTIONS" 

A.  INTRODUCTION 

This  chapter  provides  background  information 
underlying  the  Government's  motive  to  develop  an  innovative 
approach  to  attracting  commercial  research  and  development 
(R&D)  business  through  the  use  of  Other  Transaction 
Authority  (OTA) .  This  chapter  will  also  discuss  both  the 
legislative  and  operational  intent  of  Section  845  Other 
Transactions . 

Three  recent  research  efforts  on  OTs  conducted  at  the 
Naval  Postgraduate  School  (NPS)  have  provided  much  of  the 
background  information  for  this  thesis .  Each  researcher 
focused  on  a  specific  area  of  OTA.  Howell  examined  whether 
DOD  could  use  OTs  in  lieu  of  the  standard  procurement 
system  in  applying  commercial  research  and  development 
efforts  to  military  systems.  [Ref.  7]  Slade's  research 
discussed  using  OTs  in  prototype  development.  [Ref.  8] 

Hayes'  thesis  developed  an  OT  decision  model.  [Ref.  9] 
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B.  THE  MACRO  ENVIRONMENT 

1.  Post  Cold  War 

Unlike  the  cold  war  R&D  effort  that  focused  on 
fielding  superior  weapon  systems  at  any  cost,  today's 
weapon  systems  must  be  both  technologically  superior  and 
affordable.  [Ref.  10]  Today,  DOD  is  shifting  away  from  a 
world  in  which  performance  is  the  only  consideration, 
towards  a  more  balanced  life  cycle  "cost  of  performance" 
view.  [Ref.  11]  This  fundamental  change  in  the  way  America 
develops  and  fields  weapon  systems  reflects  a  dramatic 
shift  from  the  Soviet  Union  as  the  known  adversary  to  an 
environment  in  which  there  are  diverse  unpredictable 
threats.  [Ref.  12:p.  6]  Large,  expensive  weapon  systems 
aimed  to  counter  a  known  threat  are  giving  way  to  highly 
technical,  information-based  weapon  systems  designed  to 
detect,  prosecute  and  integrate  coordinated  attacks  on  the 
agile  threats  expected  in  future  conflicts.  [Ref.  13 :p.  21] 
This  pursuit  of  high-technology  weapon  systems  requires  a 
more  flexible  acquisition  approach  than  those  used  in  the 
past.  [Ref.  7:p.  9] 

In  the  past,  performance  was  not  considered  a  tradeoff 
against  cost  in  producing  superior  weapon  systems.  [Ref. 
7:p.  9]  However,  today's  military  must  consider  cost  when 
deciding  which  technologies  and  systems  to  pursue  with 


12 


limited  resources  and  defense  dollars.  As  a  result. 
Services  are  experiencing  greater  pressure  to  improve 
efficiency  in  the  acquisition  process  to  conserve  valuable 
resources.  To  improve  efficiencies  in  the  acquisition 
process,  DOD  is  striving  to  develop  and  procure  weapon 
systems  of  tomorrow— better,  smarter,  cheaper  and  faster. 
[Ref.  14 :p.  1]  To  accomplish  this,  DOD  has  focused  on: 
promoting  acquisition  reform  initiatives  that  reduce  total 
cost  and  cycle  times;  improving  efficiency  of  internal 
processes  that  have  a  major  impact  on  cycle  time;  and 
reducing  the  time  to  develop  and  insert  new  technologies 
that  improve  war-fighting  capabilities.  [Ref.  14 :p.  7] 

2 .  The  National  Industrial  Base 

Since  the  end  of  the  cold  war,  the  defense  industrial 
base  has  undergone  significant  changes  in  the  form  of 
mergers  and  acquisitions.  While  DOD  purchases  have 
declined,  commercial  markets  have  continued  to  expand, 
reducing  DOD's  spending  role  as  a  driving  force  for 
innovation.  [Ref.  15:p.  A20]  The  Services  no  longer 
dominate  advanced  technology.  Many  highly  advanced 
"civilian"  technologies  are  being  developed,  fielded  and 
deployed  by  commercial  firms  well  in  advance  of  DOD's 


13 


requirements  for  the  same  technology.  In  fact,  according 

to  management  consultant  Robert  Spreng: 

A  significant  share  of  the  most  valuable  research 
and  development  activity  in  commercial  companies 
is  virtually  unavailable  to  the  Federal 
Government,  despite  the  potential  benefits  to 
both  parties.  [Ref.  8,  pp.3] 

Within  the  past  six  years,  Intel  Corporation,  International 

Business  Machines  (IBM) ,  DuPont,  and  General  Electric 

Company  have  either  sold  or  shut  down  their  defense  R&D 

businesses.  [Ref.  15 :p.  A20]  According  to  the  Director  of 

Defense  Reform,  OSD,  Stan  Solaway: 

Three-quarters  of  the  country's  top  75  or  so 
information-technology  companies  won't  do 
research  for  the  military.  [Ref.  15 :p.  A20] 

The  disparity  between  the  DOD's  and  the  commercial 
sector's  investment  in  R&D  has'  been  growing  wider  since  the 
end  of  the  cold  war.  [Ref.  15:p.  A20]  This  difference 
means  that  relatively  more  of  the  nation's  technological 
momentum  will  come  from  commercial  enterprises  rather  than 
from  defense  programs.  In  1971,  the  U.S.  Patent  Office 
awarded  1,271  patents  to  the  Services,  accounting  for  1.6% 
of  total  patents  issued.  In  1998,  the  figure  dropped  to 
585  or  0.4%.  [Ref.  15:p.  A20]  Unlike  the  past,  this  role 
reversal  has  made  it  increasingly  important  for  DOD  to 
partner  with  industry  to  leverage  commercial  practices  and 
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technological  advances,  and  reduce  barriers,  such  as  those 
imposed  by  the  FAR/DFARS.  [Ref.  16:pp.  3-8]  If  DOD  is  to 
exploit  advanced  technology  to  develop,  field  and  sustain 
superior  weapon  systems,  it  will  have  to  rely  increasingly 
upon  the  same  industrial  base  that  builds  commercial 
products.  [Ref.  7:p.  15] 

3.  Joint  Vision  2010 

Joint  Vision  (JV)  2010  boldly  structures  a  conceptual 
framework  for  how  our  Armed  Forces  should  approach  the 
future.  JV  2010  is  built  on  the  premise  that  modern  and 
emerging  technological  innovation  in  the  commercial  market 
place  will  leverage  our  military's  effectiveness  in 
achieving  full-spectrum  dominance.  To  achieve  this  end, 
the  Services  must  find  ways  to  integrate  emerging 
technological  advances  with  innovative  thinking.  [Ref. 

13  :p.  11]  JV  2010  advocates  harnessing  commercial 
innovation  and  exploiting  dual-use  technologies  to  maximize 
weapon  system  affordability.  [Ref.  13 :p.  32] 

In  this  era  of  budgetary  constraint,  which  threatens 
future  investment  in  defense  technology,  dual-use  and 
commercial  off-the-shelf  (COTS)  technology  is  expected  to 
decrease  costs  and  shorten  acquisition  and  development 
times.  [Ref.  17]  The  National  Security  Strategy,  along 
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with  JV  2010,  seeks  to  enhance  the  commercial  and  defense 
industrial  base  to  better  serve  defense  needs,  promote  U.S. 
economic  competitiveness,  and  provide  U.S.  industry  with 
the  benefit  of  combined  larger  markets.  [Ref.  12 :p.  29] 

4.  Commercial  Access 

As  described  in  JV  2010,  the  nation's  Science  and 
Technology  program  will  help  build  a  common  industrial  base 
using  commercial  practices,  processes,  and  products.  It 
will  develop,  wherever  possible,  technology  for  both 
military  and  commercial  (dual-use)  products  and 
applications  [Ref.  17 :p.  32].  However,  despite  this  clear 
vision  statement,  many  commercial  firms  have  avoided  doing 
business  with  the  Federal  Government  because  of  cumbersome 
requirements  and  regulations.  [Ref.  15 :p.  A20]  Businesses 
who  might  otherwise  engage  in  joint  civilian/military  R&D 
endeavors  have  hesitated  to  participate  because  of  issues 
of  intellectual  property  rights,  technical  data  rights,  and 
certified  cost  and  pricing  data,  just  to  name  a  few.  [Ref. 
7:p.  12].  Conversely,  some  military  buying  agencies  have 
been  reluctant  to  enter  into  R&D  projects  with  commercial 
firms  for  lack  of  definitive  requirements  or  structured 
agreements.  [Ref.  18 :p.  5] 
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5.  The  Need  for  an  Alternative  Approach 


Under  the  auspices  of  the  Department  of  Defense 

Science  and  Technology  Strategy,  DARPA,  DOD's  central  R&D 

organization,  has  spearheaded  efforts  to  tap  into 

commercial  advances  in  new  technology.  [Ref.  9:pp.  9-12] 

To  help  broaden  their  access  to  commercial  firms  not 

familiar  or  willing  to  conduct  business  with  the 

Government,  DARPA  sought  an  alternative  "contractual" 

approach  to  negotiating  terms  and  conditions  that  is  more 

flexible  than  the  standard  FAR  contract  or  cooperative 

agreement.  [Ref.  19:p.  35]  Grants  and  cooperative 

agreements  authorized  under  the  Federal  Grant  and 

Cooperative  Agreement  Act  (1978)  did  not  allow  DARPA  enough 

contractual  freedom  to  interact  with  high-technology 

commercial  firms.  [Ref.  20 :p.  14]  As  a  result,  DARPA 

missed  many  opportunities  to  contract  with  companies 

developing  new  technologies. 

DARPA  found  that  some  of  the  most  promising 
technical  ideas  were  found  in  small  start-up 
companies  that  were  often  made  up  of  nothing  more 
than  the  owners  of  intellectual  property,  the 
skills  of  their  principals,  and  a  few  key 
employees.  [Ref.  20 :p.  14] 

Many  of  DARPA' s  endeavors  required  consortium 
arrangements,  which  were  not  appropriate  for  standard 
contracts  or  cooperative  agreements.  Participants  needed 
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to  be  recognized  as  peers  or  co-prime  contractors  in  an 
arrangement  that  could  be  accomplished  only  through  multi¬ 
party  agreement.  [Ref.  20 :p.  14]  DARPA  believed  OTs  were 
ideal  instruments  for  these  agreements.  [Ref.  21]  The 
flexibility  and  cost-sharing  attributes  would  allow 
consortium  members  to  approach  R&D  projects  from  a 
different  standpoint.  In  the  past,  the  key  question  to 
answer  was:  "How  much  cost  estimate  can  I  sell  my 
Government  customer  that  will  reduce  my  risk  and  maximize 
my  fee?"  Now,  the  key  question  would  be:  "How  can  we  craft 
an  agreement  that  will  achieve  goals  at  the  least  possible 
cost  for  all  of  us?"  [Ref.  20:p.  38] 

The  biennial  Goldwater-Nichols  Act  Implementation 
Report  of  1988  questioned  DARPA' s  inability  to  enter  into 
cost-sharing  and  cost-recovery  agreements  to  develop  dual- 
use  technologies.  [Ref.  22 :p.  4]  The  report  recommended 
that  DARPA  be  authorized  to  enter  into  "innovative 
contractual  arrangements"  that  were  in  the  best  interest  of 
the  Government.  [Ref.  22 :p.  4] 

C.  LEGISLATIVE  INTENT 

In  1989,  Congress  enacted  legislation  under  Section 
2371  of  Title  10,  United  States  Code  (U.S.C),  giving  DARPA 
the  authority  to  enter  cooperative  agreements  and  "other 
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transactions"  for  advanced  research  projects.  [Ref.  18 :p. 
2]  Cooperative  agreements  are  assistance  instruments, 
rather  than  acquisition  instruments ,  that  are  to  be  used 
only  when  the  principal  objective  of  a  transaction  is  to 
accomplish  a  public  purpose  of  support  or  stimulation 
authorized  by  Federal  statute.  [Ref.  23]  The  statutory 
criterion  for  choosing  a  cooperative  agreement  is  that 
substantial  involvement  is  expected  between  the  executive 
agency  and  the  state  or  local  government,  or  other 
recipient,  when  carrying  out  the  activity  contemplated  in 
the  agreement . 

The  language,  codified  in  10  U.S.C.  2371,  did  not 
specifically  define  "other  transactions."  This  gave  DARPA 
the  flexibility  it  desired  to  deal  with  unique  situations 
encountered  when  fostering  advanced  technology,  especially 
dual-use  technology.  [Ref.  l:p.  2]  Although  OT  was  not 
specifically  defined,  the  Congress  clearly  intended  this 
authority  to  support  research  performed  by  commercial 
firms . 

The  Secretary  of  Defense,  in  carrying  out 
advanced  research  projects  through  the  (Defense) 
Advanced  Research  Projects  Agency,  may  enter  into 
cooperative  agreements  and  other  transactions 
with  any  person,  any  agency  or  instrumentality  of 
the  United  States,  any  unit  of  State  or  local 
Government,  any  educational  institution,  and  any 
other  entity.  [10  U.S.  Code  2371] 
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Congress  originally  limited  other  transaction  authority 
(OTA)  to  a  two-year  trial  period  and  restricted  its  use  to 
situations  where  standard  contracts,  cooperative 
agreements,  or  grants  were  not  feasible  or  appropriate.  In 
1991,  Congress  made  the  authority  permanent  and  amended  10 
U.S.C.  2371,  permitting  the  Services  to  use  the  authority. 
However,  up  until  late  1993,  the  Services  were  required  to 
obtain  the  Deputy  Secretary  of  Defense's  approval  to  use 
the  authority;  the  result  was  that  System  Commands  did  not 
use  OTs.  [Ref.  l:p.  2] 

In  the  Fiscal  Year  1994  Defense  Authorization  Bill, 
Section  845,  Congress  amended  OT  authority  for  DARPA  to 
include  prototypes  that  are  "directly  relevant"  to  weapons 
or  weapon  systems  "proposed  to  be  acquired."  [Ref.  8:p.  11] 
Commonly  referred  to  as  "Section  845  prototype  authority," 
it  was  originally  limited  to  three  years.  It  was  extended 
an  additional  three  years  by  Section  804  of  the  National 
Defense  Authorization  Act  for  Fiscal  Year  1997,  Public  Law 
104-201,  and  made  available  to  the  Secretary  of  a  military 
department  or  any  other  official  designated  by  the 
Secretary  of  Defense. 

The  Dual-Use  Application  Programs  (DUAP)  and  Commercial 
Operations  and  Support  Savings  Initiative  (COSSI),  under 
the  Department  of  Defense's  Science  and  Technology 
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Strategy,  proved  to  be  an  opportune  platform  from  which 
DARPA  and  the  Services  could  gain  access  to  commercial 
technology,  using  OT  as  its  vehicle.  [Ref.  24 :p.  ii]  COSSI 
was  created  by  Public  Law  104-206  of  the  1997  Defense 
Appropriation  Act  as  part  of  DARPA' s  Dual-Use  Application 
Program.  [Ref.  25 :p.  1]  It  was  designed  to  reduce 
operating  and  support  costs  by  inserting  commercial 
technology  and  components  into  fielded  military  systems. 

The  OTA  was  considered  the  best  tool  for  achieving  COSSI' s 
goals  of  seeking  cost-saving  ideas  from  industry, 
experimenting  with  a  new  way  of  doing  business,  taking 
advantage  of  commercial  technological  advances,  getting 
contractors  to  cost-share,  and  encouraging  non- traditional 
suppliers  to  participate  in  the  program.  [Ref.  26] 

The  National  Defense  Authorization  Act  for  Fiscal  Year 
1999,  Public  Law  105-261,  Section  241,  extended  Section  845 
prototype  authority  through  September  2001.  [Ref.  24 :p.  ii] 
Most  recently,  the  National  Defense  Authorization  Act  for 
Fiscal  Year  2000,  Public  Law  106-79,  Section  801,  amends 
Section  845  prototype  authority  by  granting  the  Comptroller 
General  authority  to  conduct  audits  subject  to  the 
following:  [Ref.  27] 
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Each  agreement  entered  into  by  an  official  stated 
in  subsection  (a)  to  carry  out  a  project  in 
excess  of  $5,000,000  shall  include  a  clause  that 
provides  for  the  Comptroller  General,  in  the 
discretion  of  the  Comptroller  General,  to  examine 
the  records  of  any  party  to  the  agreement  or  any 
entity  that  participates  in  the  performance  of 
the  agreement.  [P.L. 106-79,  Title  VIII, 
sec .801 (c) ] 


D.  OPERATIONAL  INTENT 

1.  Freedom  from  Mandates,  Statutes,  and  Regulations 

As  previously  mentioned.  Congress  made  OT  authority 
permanent  in  1991  and  amended  10  U.S.C.  2371  to  include 
prototype  projects  for  DARPA  in  1994  and  the  Services  in 
1997.  Recognizing  the  need  to  enhance  flexibility  and 
reduce  the  burden  of  Government -funded  science  and 
technology  contracts.  Congress  crafted  Section  845  OTs  to 
release  DARPA  and  the  Services  from  complying  with  statutes 
and  regulations  in  the  FAR/DFARS  procurement  process . 
Freedom  from  mandates,  regulations  and  statutes  has  been 
especially  instrumental  in  resolving  critical  issues  with 
commercial  industry  regarding  intellectual  property  rights 
(IPR) .  [Ref.  7:p.  20] 

2.  Attracting  Commercial  Firms 

A  key  feature  of  this  authority  is  that  most 
procurement  statutes  and  Federal  Acquisition  Regulations 
(FAR)  do  not  apply  to  OTs,  allowing  for  greater  flexibility 
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and  a  broader  business  strategy.  [Ref.  5]  This  greater 
flexibility  attracts  commercial  firms  that  normally  would 
not  do  business  with  the  Government,  thus  expanding  the 
defense  acquisition  technology  and  industrial  base.  [Ref. 

24 :p .  ii] 

Despite  improved  flexibility,  DOD  Systems  Commands  did 
not  use  OTs  until  late  1993  because  they  were  required  to 
obtain  pre-approval  from  the  Deputy  Secretary  of  Defense. 
[Ref.  1 : p .  2]  Instead,  they  used  "flexible"  cooperative 
agreements  with  industry  and  only  managed  OTs  for  DARPA 
projects  using  DARPA  funds.  According  to  the  OUSD/DDP 
IPT's  Final  Report,  there  is  a  false  perception  among  the 
Services  that  cooperative  agreements  are  more  flexible  than 
OTs.  [Ref.  1:  Section  IV]  In  actuality,  cooperative 
agreements  are  very  limiting  because  they  are  intended  for 
R&D  projects  performed  by  universities  and  other  nonprofit 
organizations.  [Ref.  23] 

Attempting  to  ease  that  restriction  and  reverse  the 
cultural  resistance  and  skepticism  surrounding  the 
authority,  the  Director  for  Defense  Research  and 
Engineering  (DDR&E)  issued  interim  guidance  for  the 
Services  and  DARPA  on  using  OTs.  [Ref.  1:  Section  IV]  The 
guidance  encouraged  awarding  agencies  to  consider  the 
authority  as  an  "opportunity"  to  develop  innovative 
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approaches  to  research  and  prototype  projects.  DDR&E  issued 
two  Supplements  to  this  guidance  in  1997  and  1998, 
establishing  policy  that  encouraged  users  to  reduce 
administrative  burdens,  craft  innovative  agreements,  and  to 
seek  waivers  for  regulatory  requirements  if  necessary. 

[Ref.  1:  Section  IV] 

Since  DDR&E  issued  its  interim  guidance  and 
supplements,  Military  departments  and  defense  agencies  have 
reported  that  OTAs  enable  program  offices  to  establish 
business  relationships  that  would  be  impossible  under  a 
procurement  contract.  [Ref.  24:p.  ii]  These  relationships, 
include  awards  to  consortia;  companies  that  typically  do 
not  do  business  with  the  government;  commercial  business 
units  versus  designated  government  business  units  within  a 
parent  corporation;  educational  institutions;  and  foreign 
contractors.  They  also  permit  traditional  defense 
contractors  to  enter  business  arrangements  with  lower-tier 
contractors  who  normally  would  not  be  subcontractors  under 
FAR-type  contracts.  [Ref.  24 :p.  145]  This  enables 
traditional  defense  contractors  to  access  technology  that 
otherwise  might  not  have  been  available. 

For  example,  the  U.S.  Army  Communication-Electronics 
Command  recently  awarded  a  $1,436  million  Section  845  OTA 
to  Raytheon  TI  Systems  to  design  a  Second  Generation 
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Forward  Looking  Infrared  (FLIR)  Aviation  Demonstrator. 

[Ref.  24 :p.  140]  The  agreement,  aimed  at  developing  a 
secondary  source  of  supply  for  the  next-generation  FLIR, 
has  involved  several  different  subcontractors  who  do  not 
typically  accept  Government  business  due  to  excessive 
regulations,  IPR  constraints  and  Cost  Accounting  Standards 
(CAS)  . 

According  to  Jasper,  the  essence  of  the  OT  arrangement 
is  true  teaming  among  all  participants.  [Ref.  28 :p.  4]  In 
1995,  the  Institute  for  Defense  Analysis  (IDA)  conducted  a 
survey  of  organizations  that  participated  in  OTs  with 
DARPA.  The  IDA  findings  confirmed  the  importance  of 
building  a  collaborative  relationship  that  works. 

E.  HOW  SECTION  845  PROTOTYPE  AUTHORITY  DIFFERS  FROM 

STANDARD  CONTRACTUAL  TRANSACTIONS 

OTs  are  best  described  by  what  they  are  not.  [Ref.  5] 
They  are  not  contractual  instruments ,  cooperative 
agreements  or  grants.  Rather,  they  are  funding 
transactions  between  Government  and  commercial  firms  or 
consortiums .  [Ref.  29]  Whereas  FAR- type  contracts  are  best 
suited  for  the  buyer  seller  relationship,  OTs  involve  R&D, 
stimulation,  support,  and  close  cooperation  among 
participants.  [Ref.  28 :p.  6]  OTs  are  relieved  from  many 
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mandates,  statutes  and  regulations,  which  provides 
negotiators  the  flexibility  to  craft  agreements  that  can 
address  a  broad  spectrum  of  issues.  [Ref.  4]  Examples  of 
some  of  these  regulations  are  the  Armed  Services 
Procurement  Act,  Competition  in  Contracting  (CICA) , 

Contract  Disputes  Act,  Public  Law  85-804  Extraordinary 
Contractual  Relief,  Anti-Kickback  Act,  Procurement 
Integrity  Act,  Buy  American  Act,  Service  Contract  Act,  the 
bid  protest  system,  and  others.  [Ref.  28 :p.  3]  It  is 
important  to  note  that  this  does  not  mean  laws  can  be 
ignored.  It  does  mean  that  flexibility  is  provided  to  come 
to  agreement  on  terms  and  conditions  that  are  of  particular 
interest  to  the  program.  [Ref.  28 :p.  3] 

Using  OTs  provides  the  opportunity  for  participants  to 
think  outside  the  box.  Contractors  can  establish  separate 
cost  centers  that  reduce  overhead.  [Ref.  28 :p.  3] 

According  to  Jasper,  proposal  preparation  and  source 
selection  times  and  cost  may  be  reduced.  Critical  issues 
include  intellectual  property  rights  (IPR) ,  patents, 
payment  terms,  cost-sharing  arrangements,  disputes,  cost 
and  accounting  standards,  just  to  name  a  few. 

All  terms  and  conditions  are  negotiated  from  a  "best 
practice"  position  outside  the  realm  of  FAR/DFARS.  This 
has  made  Section  845  OT  attractive  to  commercial  firms. 
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DARPA  and  DOD  Systems  Commands  because  it  offers  more 
flexibility  and  imposes  fewer  regulatory  requirements  than 
does  a  typical  FAR  contract.  Conditions  applicable  to  OTs 
are:  [Ref.  29] 

1.  They  are  used  only  to  carry  out  basic,  applied, 
advanced  research  and  prototype  projects ; 

2 .  They  are  funding  transactions  that  are  not 

contracts,  grants,  or  cooperative  agreements; 

3.  To  the  maximum  extent  practicable,  research  must 
not  duplicate  research  being  conducted  under 
existing  programs  carried  out  by  the  DOD; 

4 .  To  the  extent  the  Secretary  of  Defense  determines 
practicable,  the  funds  provided  by  the  Government 
should  not  exceed  the  total  amount  provided  by 
other  parties  to  the  other  transaction; 

5 .  Other  transactions  may  be  used  when  the  use  of  a 
standard  contract,  grant,  or  cooperative 
agreement  is  not  feasible  or  appropriate. 

The  conditions  of  Section  845  OTs  are  the  same  as  10  U.S.C. 

2371,  except  that  they:  [Ref.  29] 

6.  Do  not  require  contractor  cost  matching; 

7.  Do  not  require  determination  that  a  contract, 
grant  or  cooperative  agreement  is  not  feasible  or 
appropriate; 

8.  Require,  to  the  maximum  extent  practicable,  use 

of  competitive  procedures. 


The  major  differences  between  OTs  and  standard 
contractual  agreements  are  briefly  explained  below:  [Ref. 
29] 
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1.  No  DCAA  involvement.  Cost  or  pricing  data  may  be 
obtained,  but  certification  is  not  required.  The 
commercial  firm's  auditors  working  the  Section 
845  agreement  certify  invoices  for  accuracy, 
allocability,  and  allowability.  Some  agreements 
have  commissioned  independent  accounting  firms  to 
assure  the  "interests  of  all  parties  are 
protected. "  Generally  Accepted  Accounting 
Principles  (GAAP)  are  followed. 

2.  Minimal  DCMC  involvement.  DCMC  assists  issuing 
activities  in  developing  solicitation  packages 
and  agreement  terms.  [Ref.  30 :p.  3] 

3 .  Comptroller  General  has  the  discretion  to  examine 
the  records  of  any  party  to  an  agreement  in 
excess  of  $5,000,000  for  a  period  not  to  exceed 
three  years  after  the  final  payment.  [Ref.  27] 

4 .  Minimum  Socio-Economic  Clauses .  No  Small 
Business  Administration  or  "buy  American" 
provisions  in  effect. 

5.  Flexible  payment  provisions.  Invoices  can 
include  subcontractor  work- in-progress,  even  if 
not  yet  paid. 

6.  Simplified  dispute  resolution.  Disputes  do  not 
require  contractor  certification  of  claim  or 
continued  contractor  performance .  Disputes  are 
ultimately  resolved  by  a  majority  vote  of  a  board 
consisting  of  the  consortium  members  and  an 
impartial  third  party.  Disputes  must  be  raised 
within  3  months  instead  of  6  years. 

7.  Cost  Accounting  Standards  (CAS).  Contractor  must 
comply  with  the  intent  of  the  FAR  coverage,  but 
no  DCAA  involvement  is  required.  Accounting 
practices  can  be  changed  unilaterally  by  the 
contractor  during  the  course  of  the  agreement. 


F.  SUMMARY 

Arising  from  DARPA's  desire  to  tap  into  the 
commercial  market,  OTA  is  an  alternative  approach  to  FAR- 
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type  contracts.  OTs  are  characterized  by  greater 
flexibility  and  innovation  in  R&D  projects.  They  are 
intended  to  expand  the  military  industrial  market  place  by 
attracting  R&D  firms  not  used  to,  or  familiar  with, 
conducting  Federal  Government  business.  [Ref.  29]  OTs 
allow  contractors  to  consider  new  ways  of  doing  business 
and  permit  strictly  commercial  firms  access  to  DOD  projects 
without  having  to  change  their  existing  business  practices. 
[Ref.  28:p.  1] 

The  next  chapter  will  explore  performance  metrics. 
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III.  PERFORMANCE  METRICS 


A.  INTRODUCTION 

This  chapter  introduces  performance  metrics,  providing 
definition,  purpose,  attributes,  benefits  and  concerns. 

The  chapter  describes  categories  used  to  classify  metrics 
and  a  basic  sequence  for  establishing  metrics  in  an 
organization. 

The  research  effort  conducted  by  Gordon  at  the  Naval 
Postgraduate  School  (NPS)  on  Metric  Evaluation  Approach  for 
the  Defense  Acquisition  Workforce  Improvement  Act  provides 
a  more  comprehensive  discussion  of  these  issues.  Gordon's 
study  provided  much  of  the  background  information  for  this 
chapter.  [Ref.  39] 

B.  MANAGING  PERFORMANCE 

1.  Why  Measure  Performance? 

Appraising  and  controlling  operations  are  usually  the 
last  steps  in  the  management  process.  According  to  Dobler 
and  Burt,  no  matter  how  well  a  business,  program,  or 
process  is  conceived  or  organized,  a  lack  of  proper 
controls  or  performance  indicators  can  render  it 
ineffective.  [Ref.  31 :p.  671]  Executives  and  managers 
directing  the  efforts  of  an  organization  or  a  group  have  a 
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responsibility  to  know  how,  when,  and  where  to  institute  a 
wide  range  of  changes .  They  cannot  implement  these  changes 
intelligently  without  knowledge  of  the  appropriate 
information  on  which  the  changes  are  based. 

Tracking  and  measuring  performance  is  an  inherent  part 
of  monitoring,  controlling,  and  improving  a  process  or 
activity;  control  and  appraisal  are,  therefore,  essential 
to  achieving  organizational  goals  and  objectives.  If 
managers  are  to  manage  programs  and  improve  outcomes,  they 
must  measure  performance.  According  to  the  Organization 
for  Economic  Cooperation  and  Development  (OECD) ,  an 
international  public  policy  research  institute  based  in 
Paris : 

The  main  objective  of  performance  measurement  in 
public  organizations  is  to  support  better 
management  decision-making  leading  to  improved 
outcomes  for  the  community,  and  to  meet  external 
accountability  requirements.  [Ref.  32 :p.  1] 

For  this  reason,  managers  measure  performances  to  monitor, 
control,  and  improve  at  three  levels  of  performance: 
organization,  process,  and  job  performance.  [Ref.  33 :p.  26] 
Organization  measures  serve  the  needs  of  the  highest  levels 
in  the  organization:  corporate,  divisional,  and  functional. 
An  example  of  the  organizational  level  would  be  ASN 
(RD&A)'s  goal  of  reducing  Total  Ownership  Costs  to  increase 
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resources  available  for  re-capitalization  and 
modernization.  [Ref.  14]  To  achieve  this  goal,  several  of 
their  performance  measures  are  focused  on  acquisition 
reform  initiatives  aimed  at  improving  efficiencies  of 
processes  that  have  a  major  impact  on  acquisition  cycle 
times . 

Process  metrics  measure  performance  of  departments, 
product  lines,  and  classes  of  service.  In  large 
organizations,  such  as  DOD,  there  may  be  many  layers  of 
this  category.  For  example,  as  the  Head  Contracting  Agency 
for  the  Navy,  Naval  Supply  Systems  Command  (NAVSUP) 
monitors  the  performance  of  the  Afloat  Purchase  Card 
program  by  measuring  both  usage  and  delinquency  rates,  and, 
the  number  of  purchase  card  software  requests.  NAVSUP 
monitors  other  process  goals,  of  subordinate  commands, 
including  paperless  acquisition  initiatives  and 
competition. 

Job  performance  metrics  gather  basic  data.  They 
summarize  the  performance  of  specific  processes,  programs, 
functional  areas,  service  cycles,  and  persons.  [Ref.  34 :p. 
1.6]  For  example,  the  Naval  Sea  Systems  Command  (NAVSEA) 
measures  staff  workload  performance,  number  of  competitive 
awards,  claims  and  protests,  and  closely  monitors  customer 
service . 
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2.  What  are  Performance  Measures? 
a .  Defini tions 

(1)  Measurement:  "(noun),  the  act  or 
process  of  measuring;  a  data  element  having  a 
dimension,  capacity,  or  amount  of  something 
ascertained  by  measuring."  [Ref.  35 :p.  1] 

(2)  Metric:  "(noun),  a  standard  of 
measurement;  a  specialized  measurement  relating  to 
measuring  a  process,  or  process  step,  under  your  direct 
control."  [Ref.  35 :p.  1] 

(3)  Performance  Measure:  "A  generic  term 
encompassing  the  quantitative  basis  by  which  objectives  are 
established  and  performance  is  assessed  and  gauged. 
Performance  measures  include  performance  objectives  and 
criteria,  performance  indicators,  and  any  other  means  to 
evaluate  the  success  in  achieving  a  specified  goal."  [Ref. 
34 :p.  1.30] 

The  Metrics  Handbook  (METRICS),  by  U.S.  Air  Force 

Systems  Command,  provides  a  user-friendly  working 

definition  of  metrics: 

Metrics  are  nothing  more  than  meaningful 
measures.  For  a  measure  to  be  meaningful, 
however,  it  must  present  data  that  allow  us  to 
take  action.  Metrics  foster  process  understanding 
and  motivate  action  to  continually  improve  the 
way  we  do  business.  This  is  distinguished  from 
measurement,  in  that,  measurement  does  not 
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necessarily  result  in  process  improvement.  Good 
metrics  always  will.  [Ref.  35 :p.  1] 

b.  The  Purpose  of  Performance  Measures 

Performance  measures  are  usually  approached  from 
two  directions:  problem  prevention  and  monitoring  systems. 
[Ref.  31 :p.  671]  Regardless  of  the  approach  taken, 
establishing  a  realistic  standard  of  performance  that  is 
accurate  and  accepted  is  the  most  difficult  task.  When 
properly  designed,  metrics  can  help  managers  accomplish 
organizational  goals  and  objectives.  [Ref.  31:pp.  674-675] 
Ideally,  they  help  monitor  requirements ,  predict  outcomes, 
track  progress,  and  lead  to  a  better  overall  understanding 
of  risk.  [Ref.  36:p.  7] 

Performance  measures  quantitatively  tell  us 
something  important  about  our  products,  services,  and 
processes  by  providing  a  single-dimensional  or 
multidimensional  quantitative  measure.  [Ref.  34 :p.  1.4]  An 
example  of  a  single-dimensional  performance  measure  would 
be:  "Question:  How  many  contractors  bid  for  the  project? 
Answer: 10."  Single-dimension  measures  usually  represent 
fundamental  measures  of  some  process  or  product.  [Ref. 

34 :p.  1.4]  A  multidimensional  performance  measure  is 
expressed  as  a  ratio  of  two  or  more  units  of  measure.  An 
example  of  a  multidimensional  performance  measure  would  be: 
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"The  number  of  on-time  deliveries  per  total  number  of 
deliveries."  Performance  measures  expressed  this  way 
almost  always  convey  more  information  than  do  single¬ 
dimension  measures.  [Ref.  34:p.  1.4] 

Common  purposes  for  measuring  performance 
according  to  Training  Resources  and  Data  Exchange  (TRADE) 
include  the  following.  [Ref.  34 :p.  1.7] 

a .  Management  Assessment 

The  purposes  of  management  assessment  are: 

1.  To  know  what  is  going  on  in  the  organization. 

2.  To  aid  in  making  decisions  regarding  resources, 
plans,  policies,  schedules,  and  structures. 

3 .  To  evaluate  whether  the  organization  is  meeting 
value-added  objectives. 

4 .  To  evaluate  whether  the  organization  is  being 
effective  and  efficient 

5 .  To  communicate  performance  expectations  to 
subordinates . 

b.  Self  Assessment 

The  purposes  of  self  assessment  are: 

1.  To  assess  how  well  a  process  is  doing,  including 

improvements  that  have  been  made . 

2.  To  identify  performance  that  should  be  rewarded. 

c.  Control 

The  purposes  of  control  are: 

1.  To  identify  performance  disparities  that  should 

be  analyzed  and  eliminated. 
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2 .  To  provide  feedback  that  compares  performance  to 
a  standard. 

d.  Continuous  Improvement 

The  purposes  of  continuous  improvement  are: 

1.  To  help  reduce  process  variation. 

2.  To  identify  defect  sources,  process  trends, 
defect  prevention,  and  to  determine  process 
efficiency  and  effectiveness. 

3.  Attributes  of  Metrics 

Measuring  the  right  variables  has  a  lot  to  do  with  the 
likelihood  of  an  organization's  success.  [Ref.  33 :p.  26] 
According  to  Brown,  to  be  a  useful  measure  of  performance, 
a  metric  should:  [Ref.  37 :p.  2] 

1.  Be  meaningful  to  the  customer. 

2.  Relate  to  organizational  goals. 

3.  Be  simple,  understandable,  logical,  and 

repeatable . 

4 .  Show  a  trend . 

5.  Be  clearly  defined. 

6.  Be  economical  to  collect. 

7.  Be  timely. 

8.  Drive  the  appropriate  action. 

Brown  also  advocates  that  metrics  should  start  at  the  top 
and  flow  down  to  all  levels  in  the  organization;  and,  he 
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adds,  whenever  possible,  they  should  be  combined  into  a 
single  index  to  better  assess  overall  performance.  In 
addition,  metrics  should  have  targets  or  goals  based  on 
research  rather  than  on  arbitrary  numbers.  [Ref.  37:p.  3] 

4.  Measurement  Concerns 

The  use  of  performance  measures  is  hardly  new. 
Organizations  have  been  measuring  costs,  quality,  quantity, 
cycle  time,  efficiency,  productivity,  etc.,  and  processes 
for  as  long  as  ways  to  measure  those  things  have  existed. 
[Ref.  37:p.  2]  According  to  Brown,  many  buying 
organizations  currently  measure  the  same  bottom-line- 
oriented  performance  metrics  they  measured  twenty  or  more 
years  ago.  [Ref.  37:p.  3]  What  is  new,  though,  is  having 
those  who  do  the  work  determine  some  of  what  should  be 
measured  so  that  they  might  better  control,  understand,  and 
improve  what  they  do.  [Ref.  34 :p.  1.1] 

5.  Using  Measures  Effectively 

Effective  use  of  measures  means  not  only  establishing 
them,  but  also  using  measures  properly.  According  to 
Brown,  the  maximum  number  of  metrics  any  organization 
should  employ  is  20.  [Ref.  37 :p.  4] 

Organizations  should  concentrate  on  measuring  the 

vital  few  key  variables  rather  than  the  trivial 
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many.  The  key  to  defining  a  successful  set  of 
metrics  is  paring  down  your  database  to  the  vital 
few  metrics  that  are  easy  to  monitor  and  directly 
linked  to  key  business  drivers.  [Ref.  37:p.  4] 

Rumler  and  Brache  developed  the  following  three-step 
sequence  for  establishing  measures  in  an  organization. 

[Ref.  38 :p.  142]  First,  the  organization  must  identify  the 
most  significant  outputs  of  the  organization,  process,  or 
job— i.e.,  measure  only  what  is  important.  Second,  it  must 
identify  the  "critical  dimensions"  of  performance  for  each 
of  these  outputs.  Critical  dimensions  should  be  derived 
from  the  needs  of  the  internal  and  external  customers  who 
receive  the  outputs  and  from  the  needs  of  the  organization. 
And,  finally,  the  organization  must  involve  employees  in 
designing  and  implementing  metrics,  focusing  on  developing 
measures  for  each  critical  dimension.  For  example,  if 
"attracting  non-traditional  defense  firms"  is  a  critical 
dimension,  then  one  or  more  measures  should  answer  this 
question:  "What  indicators  will  tell  us  if  Section  845  OTs 

are  attracting  non-traditional  defense  firms?" 

Selecting  the  right  metrics  or  measures  is  much  more 
than  deciding  what  to  measure.  It  is  a  key  part  of  an 
overall  strategy  for  success.  [Ref.  37 :p.  8]  According  to 
Brown: 
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What  is  difficult  is  learning  to  measure  the 
right  things  and  ignoring  other  less  useful, 
though  interesting,  data  that  do  not  contribute 
to  your  organization's  success.  Select  the  wrong 
performance  metrics  and  you  may  not  achieve  your 
organization's  goals,  although  all  measurements 
indicate  that  you  are  healthy.  Coming  up  with  a 
good  precise  set  of  metrics  is  difficult  and  may 
require  extensive  research.  [Ref.  37 :p.  8] 


Selecting  the  right  performance  measurement  isn't 
management's  only  challenge.  Brinkerhoff  and  Dressier 
addressed  four  measurement  concerns  when  constructing 
metrics:  validity,  reliability,  bias,  and  judgmental  vs. 
non j udgmental .  [Ref.  39:pp.  18-22] 

a.  Validity 

Validity  is  an  important  measurement  factor  that 
refers  to  the  relationship  between  what  is  measured  and  the 
measurement's  objective.  [Ref.  39:p.  19]  Validity  is 
established  when  the  metric  is  related  to  an  event  or 
object  that  can  be  controlled  or  manipulated.  [Ref.  40 :p. 
38] 

b.  Reliability 

Reliability  refers  to  consistency,  accuracy  and 
precision  in  measurement.  [Ref.  39:p.  19]  Consistency  is 
the  metric's  ability  to  produce  the  same  results 
repeatedly.  Accuracy  involves  the  degree  to  which  the 
measure  reflects  reality.  Precision  is  the  closeness  of 
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repeated  measurements  to  their  mean  value.  [Ref.  34 :p.  13] 
Figure  3.1  depicts  the  relationship  between  precision  and 
accuracy. 


c.  Bias 

A  bias  in  a  measurement  system  is  best  described 
as  a  tendency  or  inclination  of  outlook  that  is  a 
troublesome  source  of  error  in  sensing.  [Ref.  34 :p.  13]  In 
this  context,  "sensing"  refers  to  the  measuring  device 
(often  human)  that  can  detect  some  phenomenon.  Bias  is  a 
serious  measurement  concern  that  must  be  avoided;  however, 
it  is  hard  to  avoid  and  is  often  prevalent  in  surveys  and 
customer  questionnaires.  [Ref.  39 :p.  21] 

d.  Judgmental  vs.  Nonjudgmental 

Another  measurement  concern  regards  the  type  of 
measure  selected  to  evaluate  performance— judgmental  or 
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non judgmental .  According  to.  Gordon,  judgmental  measures 
typically  deal  in  abstractions,  while  nonjudgmental 
measures  are  based  on  tangible  data.  [Ref.  39 :p.  21] 

Judgmental  measures  require  discretion  or 
judgement  by  the  person  taking  the  measurement.  This 
involves  collecting  information,  weighing  its  value,  and 
using  it  to  make  a  statement.  Arguments  against  judgmental 
measures  assert  that  they  fail  to  adequately  measure 
perf ormance .  [Ref.  39 :p.  22]  Another  significant  weakness 
of  judgmental  measures  is  grader  bias,  both  inadvertent  and 
intentional.  For  example,  since  a  survey  is  prone  to  both 
researcher  and  respondent  bias,  it  would  not  be  considered 
an  objective  measure. 

A  nonjudgmental  measure  is  definitive  in  nature, 
countable  or  quantifiable  in  some  way.  [Ref.  39 :p.  22] 
Examples  are  time,  cost,  quantity,  and  so  on,  all  of  which 
are  generally  considered  objective.  According  to  Landy  and 
Farr,  " [t]he  implication  is  that  they  are  not  open  to 
interpretation,  that  they  are  unambiguous  and  reliable." 
[Ref.  41:p.  56]  Whenever  possible,  nonjudgmental  measures 
should  be  used  to  measure  performance. 
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e .  Measurement  Paradigms 

It  is  also  important  to  recognize  organizational 
paradigms  that  hinder  the  ability  to  accurately  track  and 
measure  performance.  [Ref.  42:pp.  58-60]  Sink  and  Tuttle 
identified  several  barriers  to  implementing  measurements 
that  are  dysfunctional  in  an  organization: 

1.  Measurement  is  threatening.  Workers  often  fear 
that  new  information  about  a  process  they  perform 
or  oversee  may  be  used  against  them.  [Ref.  42 :p. 
58] 

2 .  Measurement  has  to  be  precise .  Sink  and  Tuttle 
argue  that  performance  measurement  "does  not  have 
to  be  as  precise  as  the  measurement  in  a 
laboratory  to  be  useful."  [Ref.  42 :p.  59] 

3.  Single  indicator  focus.  When  single  indicators 
are  used,  there  is  a  tendency  to  overreact  to 
measurement  results.  An  organization's  overall 
performance  cannot  be  adequately  explained  or 
measured  by  a  single  indicator.  [Ref.  42 :p.  59] 


6.  Classification  of  Metrics 

TRADE  classifies  performance  metrics  using  six 
categories  of  measurement  (effectiveness,  efficiency, 
quality,  timeliness,  productivity,  and  safety).  [Ref.  34:p. 
1.5]  For  the  purpose  of  this  research,  the  author  will 
exclude  the  "safety"  category  since  it  does  not  pertain  to 
measurement  of  other  transactions.  In  its  place,  the 
author  adds  the  more  relevant  category,  "financial."  Most 
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performance  measures  can  be  grouped  into  one  of  the 
following  six  general  categories: 

1.  Effectiveness:  A  process  characteristic  indicating 
the  degree  to  which  the  process  output  conforms  to 
the  requirement . 

2.  Efficiency:  A  process  characteristic  indicating  the 
degree  to  which  the  process  produces  the  required 
output  at  a  minimum  resource  cost . 

3.  Quality:  The  degree  to  which  a  product  or  service 
meets  customer  requirements  and  expectations. 

4.  Timeliness:  A  measurement  of  whether  or  not  a  unit 
of  work  was  done  correctly  and  on  time .  Criteria 
must  be  established  to  define  what  constitutes 
timeliness  for  a  given  unit  of  work. 

5.  Productivity:  The  value  added  by  the  process,  labor 
or  capital  consumed. 

6.  Financial:  The  dollar  amount  to  perform  a  task  or 
produce  an  output. 


Each  of  these  six  categories  measures  performance 
rather  than  compliance.  The  difference  between  performance 
and  compliance  measures  is  that  compliance  measures 
evaluate  whether  something  was  accomplished,  while 
performance  measures  evaluate  how  well  something  was 
accomplished.  Measuring  compliance  is  a  much  simpler, 
single-dimensional  process.  A  compliance  metric  is  often 
referred  to  as  a  "YES/NO"  or  "GO/NO  GO"  metric.  [Ref.  33 :p. 
26]  These  six  categories  are  shown  in  the  first  column  of 
Table  3.1.  The  second  column  describes  what  is  being 
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measured;  and  the  third  column  gives  an  example  of  a 
metric . 


Measure  of... 

Measures... 

Example... 

Efficiency 

Ability  of  an  organization  to 
perform  a  task 

Quantity  of  over -aged 
contracts  for  closure 

Effectiveness 

Ability  of  an  organization  to 
plan  for  output  from  its 
processes 

Extent  of  commercial 
items  procured 

Timeliness 

Whether  a  unit  of  work  was 
completed  on  time 

Purchase  order  cycle 
time 

Quality 

Whether  a  unit  of  work  was 
completed  correctly 

Customer  satisfaction 
rating 

Productivity 

The  amount  of  a  resource  used 
to  produce  a  unit  of  work 

Awards  and 
Modifications  issued 

Financial 

The  amount  of  dollars  used  to 
perform  a  task 

Cost  to  Manage 
Contracts 

Table  3.1.  Classification  of  Performance  Metrics.  [Ref.  34:p.  24] 


Research  found  that  focusing  performance  measures  on 
one  class  or  category  (e.g.,  quality  or  financial)  is  not 
the  answer.  [Ref.  37 :p.  6]  Rather,  the  key  is  to  have  a 
balanced  approach  covering  several  categories .  The  Navy 
Inventory  Control  Point  (NAVICP)  provides  a  good  example. 

As  a  buying  organization  that  provides  supply  support  for 
weapon  systems,  they  are  primarily  concerned  with  ensuring 
parts  are  available,  on  demand,  for  their  customer  base. 

As  such,  they  have  focused  their  contract  performance 
measures  on  several  areas  to  capture  both  the  pre-award  and 
post-award  processes.  [Ref.  43]  For  the  pre-award 
processes,  NAVICP  monitors  timeliness  by  measuring 
Procurement  Administration  Lead-time;  productivity,  by 
measuring  the  number  of  contracts  that  are  in  work- in- 
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progress;  and  financial,  by  monitoring  the  counts  and 
dollar  value  of  pre-award  procurements .  This  balanced 
approach,  across  three  categories,  helps  NAVICP  managers 
monitor  one  of  the  most  significant  outputs  of  their 
organization,  getting  requirements  on  contract. 

The  Defense  Contract  Administration  Command  (DCMC) 
provides  another  example.  Their  focus  is  on  the  post-award 
process.  To  assist  managers  and  customers  in  monitoring 
contract  performance,  many  of  DCMC's  performance  measures 
monitor  timeliness  and  financial  concerns.  Examples  of 
DCMC's  measures  of  post-award  timeliness  include  delivery- 
forecast  timeliness,  contract  closeout  cycle  time,  percent 
schedules  on  time,  and  schedule  slippage  of  major  programs. 

Classifying  metrics  into  relevant  categories  that 
identify  a  buying  organization's  critical  dimensions  and 
outputs  is  an  important  sequence  in  defining  a  successful 
set  of  metrics.  Further  analysis  of  metric  classifications 
is  beyond  the  scope  of  this  thesis,  but  may  warrant  further 
research. 

C.  SUMMARY 

This  chapter  provided  background  information  on 
metrics,  including  definition,  purpose,  benefits,  concerns, 
and  classification  categories. 
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In  establishing  metrics,  the  organization  must  remain 
focused  on  its  most  significant  outputs  and  "critical 
dimensions"  of  performance.  The  key  is  to  develop  a  few 
metrics  that  are  easy  to  monitor  and  directly  linked  to  key 
business  drivers.  The  next  chapter  will  present  and 
analyze  contract  management  metrics  gathered  from  various 
buying  organizations. 
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XV.  CONTRACT  METRICS  AND  ANALYSIS 


A.  INTRODUCTION 

The  information  presented  in  Chapters  IV  and  V  was 
gathered  through  surveys  and  telephone  interviews  with 
acquisition  professionals  from  the  Office  of  Under 
Secretary  of  Defense,  Acquisition,  Technology  and  Logistics 
(OUSD  [AT&L] ) ,  Assistant  Secretary  of  the  Navy,  Research, 
Development  and  Acquisition  (ASN  [RD&A] ) ,  Naval  Supply 
Systems  Command  (NAVSUP) ,  Defense  Contract  Management 
Command  (DCMC) ,  Defense  Logistics  Agency  (DLA) ,  National 
Aeronautics  and  Space  Administration  (NASA) ,  Department  of 
Energy  (DOE) ,  Defense  Advanced  Research  Projects  Agency 
(DARPA) ,  Air  Force  Research  Laboratory,  various  Department 
of  Defense  (DOD)  buying  commands,  and  commercial  firms. 

All  Government  respondents  were  at  least  at  the  GS-14  level 
and  were  assured  of  anonymity. 

A  survey  of  Contract  Management  Metrics,  hereafter 
referred  to  as  the  first  survey,  was  conducted  to  obtain 
current  information  on  contract  metrics  employed  by  various 
government  and  civilian  organizations.  This  chapter  will 
describe  the  methodology  of  the  first  survey,  present  a 
summary  of  data,  and  categorize  and  analyze  the  responses. 
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Chapter  V  will  describe  the  methodology  used  for  the 
survey  on  Section  845  Other  Transaction  (OT)  Performance 
Metrics,  hereafter  referred  to  as  the  second  survey.  The 
chapter  also  will  present  a  summary  of  the  data  and  analyze 
the  responses.  The  second  survey  had  two  objectives:  to 
determine  the  extent  to  which  metrics  used  to  manage 
standard  contract  performance  are  appropriate  to  manage 
Section  845  OTs;  and  to  determine  the  extent  to  which  new 
metrics  can  be  developed  that  are  appropriate  for  measuring 
the  use  and  value  of  OTs. 

B.  A  SURVEY  OF  CONTRACT  MANAGEMENT  METRICS 

1 .  Methodology 

The  first  survey  was  conducted  to  obtain  current 
information  on  contract  management  metrics  used  by  various 
organizations.  The  survey  was  not  intended  to  catalog  the 
entire  population  of  contract  management  metrics  available, 
but,  rather,  to  identify  metrics  monitored  most  closely  by 
senior  acquisition  managers. 

Fifteen  organizations  were  surveyed.  The  author  sought 
information  from  six  Navy,  one  Air  Force,  and  two  DLA 
commands,  three  Defense  Research  Laboratories,  NASA  and 
DOE.  In  addition,  to  gain  an  industry  perspective,  the 
author  obtained  a  listing  of  Standard  Cross -Industry 


50 


Procurement  Benchmarks,  found  in  1997  by  researchers  H.  E. 
Fearon  and  B.  Bales,  from  the  Center  for  Advanced 
Purchasing  Studies.  [Ref.  44]  All  but  two  organizations 
responded  to  the  request  for  information— an  87  percent 
response  rate. 

Based  on  the  literature  review  and  organization  input, 


the  author  applied  Training  Resources  and  Data  Exchange's 
(TRADE)  metric  classification  scheme  to  the  base  population 
of  contract  management  metrics  identified  in  the  first 
survey  to  analyze  the  results  (Figure  4.1).  A  detailed 
review  of  TRADE'S  metric  classification  scheme  is  beyond 


the  scope  of  this  thesis;  however,  Chapter  III  outlines  the 
highlights . 


Figure  4.1.  Chapter  IV  Methodology.  [Source:  Developed  by  Researcher] 


As  Chapter  V  will  show,  the  researcher  selected,  from 
the  population  of  contract  management  metrics,  25  metrics 
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subjectively  determined  to  be  most  relevant  to  measuring 
OTs.  This  subset  of  the  first  survey  serves  as  a  base 
population  for  the  study's  second  survey. 

2 .  Subjectivity 

Classification  of  the  metrics  identified  in  the  first 
survey  was  a  subjective  process.  Each  of  the  six  categories 
of  performance  measures  (effectiveness,  efficiency, 
quality,  timeliness,  productivity,  and  financial)  possesses 
some  degree  of  subjectivity.  While  the  differences  between 
categories  was  clear,  they  were  not  always  obvious 
(mutually  exclusive)  and  require  intuitive  interpretation. 
This  problem  was  encountered  most  often  when  classifying 
efficiency,  timeliness  and  productivity. 

For  example,  the  majority  of  Space  and  Naval  Warfare 
Systems  Command's  (SPAWAR)  contract  metrics  were  classified 
as  measures  of  efficiency  since  they  were  oriented  towards 
measuring  the  benefits  of  electronic  commerce  and 
procurement  automation.  One  SPAWAR  metric  tracked  the 
number  of  awards  and  modifications  transmitted 
electronically.  From  the  researcher's  perspective,  data 
transmitted  electronically  may  indeed  measure  efficiency; 
however,  without  knowing  the  objective  of  the  metric,  it  is 
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unclear  whether  the  measure  is  workload-  (productivity) , 
staff-  (efficiency)  or  time-related. 

Also,  this  research  found  that  many  of  SPAWAR's 
efficiency  metrics  resembled  other  organization's 
productivity  and  timeliness  measures.  For  example,  NAVSEA 
and  NASA  track  the  number  of  contract  awards  and 
modifications  as  a  single-dimension  measure  of 
productivity.  This  information  is  versatile  and  can  also  be 
combined  to  form  multi -dimens ion  measures  applicable  to 
financial,  effectiveness,  and  efficiency  metric  categories. 
In  the  researcher's  opinion,  management  should  distinguish 
between  the  different  categories  and  provide  a  direct  link 
from  measurement  category,  to  metric,  to  a  critical 
dimension  or  output  of  the  organization. 

3.  Survey  One  Results 

The  survey  asked  for  the  following  information:  Please 
provide  a  detailed  listing  of  contract  management 
performance  measures  or  metrics  used  by  senior  level 
mangers  at  your  organization. 

a.  Data  Presentation  and  Discussion 

This  question  obtained  current  information  on 
contract  management  metrics  used  by  various  buying 
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organizations.  All  but  two  organizations  responded  to  the 
request  for  information,  providing  an  87  percent  response 
rate.  Survey  results  indicated  twelve  of  thirteen  (92%) 
respondents  used  metrics  to  measure  the  performance  of 
their  contracting  organizations. 

The  research  results  indicated  that  DARPA  does 
not  measure  the  performance  of  its  contract  organization. 
Four  respondents,  DCMC,  DLA,  NASA,  and  DOE  provided  a 
formal  list  of  contract  management  metrics.  DCMC,  DOE,  and 
NASA  metric  listings  included  title,  definition, 
measurement  objective,  computation  methodology,  data 
source,  and  frequency  of  measure.  Organizations  that  were 
not  able  to  provide  formal  metric  listings  provided 
information  via  copies  of  presentation  slides, 
spreadsheets,  or  informal  correspondence.  Several 
respondents,  interviewed  by  telephone,  noted  their 
organizations  were  in  the  process  of  reviewing  contract 
performance  metrics.  A  listing  of  contract  management 
metrics  used  by  organizations  that  responded  to  the  first 
survey  is  presented  in  Appendix  A. 
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Figure  4.2.  Contract  Management  Performance  Metrics  used  by 
respondents.  [Source:  Developed  by  Researcher] 


Survey  results  identified  401  metrics  used  to 
measure  the  performance  of  procurement  organizations 
(Figure  4.2).  DCMC  employs  154  metrics  designed  to  be  used 
at  all  levels  of  the  organization.  DCMC's  metrics  are 
based  on  services  it  provides  to  its  customers  and  are  used 
to  facilitate  the  organization's  Business  Plan  and  Mission 
Management  Reviews.  [Ref.  45]  Since  DCMC's  primary  mission 
is  contract  administration,  this  research  focuses  on 
metrics  common  to  this  functional  area  (e.g.,  contract 
closeout,  overage  undef initized  contract  actions, 
negotiation  cycle  times,  etc.).  Many  other  metrics  were 
used  to  measure  internal  management  processes,  special 
initiatives,  unfair  labor  practices,  and  plant  and 
manufacturing  clearances.  These  metrics  were  not  relevant 
to  OTs  and,  therefore,  were  not  exportable.  Consequently, 
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130  DCMC  metrics  were  found  inappropriate  for  the  purpose 
of  this  study. 

DLA  manages  approximately  four  million  consumable 
items  valued  in  excess  of  $9.8  billion.  DLA  tracks  35 
metrics  oriented  towards  commodity  management,  forecast 
accuracy,  inventory,  space  utilization,  costs,  customer 
service,  and  packaging.  [Ref.  75]  Based  on  their  nature, 
the  majority  of  these  measures  were  not  considered 
exportable  to  the  R&D  environment.  Consequently,  30  DLA 
metrics  were  down-selected  and  found  inappropriate  for  the 
purpose  of  this  study.  Subsequent  research  focused  on  the 
remaining  241  contract  management  metrics. 

Based  on  the  literature  review,  it  appears  that 
DOD  organizations  did  not  measure  the  use  and  value  of  OTs . 
All  respondents,  including  those  who  did  not  provide 
information,  confirmed  that  their  organizations  were  not 
using  metrics  to  measure  the  performance  of  OTs. 

b.  Analysis  of  the  survey  result 

A  comparison  of  performance  metrics  reveals 
procurement  organizations  measure  similar  contract 
management  processes.  Despite  this  apparent  commonality, 
fewer  than  ten  percent  of  the  metrics  identified  shared  the 
same  title. 
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The  literature  review  performed  earlier  in  this 
research  effort  indicated  that  many  buying  organizations 
use  the  same  metrics  they  measured  twenty  or  more  years 
ago.  Twelve  of  thirteen  respondents  (92%)  confirmed  this 
expectation  as  valid.  Only  four  organizations  (NASA,  DOE, 
NAVSUP ,  and  DCMC)  indicated  that  they  had  recently  added  at 
least  one  new  contract  management  metric  to  their  list  of 
performance  measures.  Metrics  considered  "new"  by  one 
organization  were,  in  many  cases,  considered  standard  by 
others.  For  example,  NASA  recently  added  procurement 
workload,  cost-to-manage,  and  customer  service  measures  to 
its  list  of  performance  metrics.  NAVSUP,  NAVSEA,  and  DCMC 
routinely  monitor  all  three  of  these  measures . 

Respondents  were  expected  to  provide  a  detailed 
list  of  their  organization's  contract  metrics,  including 
title,  definition,  measurement  objective,  data  source, 
computation  method,  and  frequency  of  measure.  Although  all 
organizations  used  computer  database  systems  to  gather  and 
assimilate  performance  measurement  data,  nine  of  thirteen 
organizations  were  unable  to  provide  a  formal  listing  of 
contract  metrics.  The  four  organizations  that  provided  a 
listing  had  recently  conducted  a  comprehensive  review  of 
their  performance  measurement  systems.  This  analysis 
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indicates  that  70  percent  of  the  responding  organizations 

were  unable  to  produce  a  formal  metric  listing. 

Several  respondents  interviewed  by  telephone 

revealed  their  concerns  with  performance  metrics. 

Representative  comments  received  included: 

We  do  not  maintain  a  detailed  listing  of  contract 
metrics.  We  do,  however,  measure  a  lot. 
Unfortunately,  data  seem  meaningless  and  are 
often  perceived  as  a  burden  to  collect.  Metrics 
need  to  be  meaningful  in  order  to  be  accepted. 

Our  organization  doesn't  have  a  formal  listing 
per  se,  but  we  do  track  numerous  contract 
performance  metrics.  Perceptions  in  the  office 
are  mixed  as  to  whether  or  not  our  measures  are 
truly  useful.  Are  we  measuring  just  to  measure, 
or  for  the  purpose  of  managing?  Different  folks 
will  give  you  different  answers. 

Although  we  do  not  maintain  a  listing  of  contract 
metrics,  I  do  believe  it  would  be  beneficial  to 
our  organization.  Their  current  use  and 
effectiveness  [metrics]  are  spotty.  What  we  need 
to  do  is  to  move  towards  a  leaner,  more 
meaningful  set  of  metrics. 


The  researcher  anticipated  that  the  metrics 
provided  would  be  identifiable  by  category  (i.e., 
effectiveness,  efficiency,  quality,  timeliness, 
productivity,  and  financial) .  Metrics  provided  by  DCMC  and 
NASA  depicted  the  classification  category  of  the  metric. 
[Ref.  45]  However,  these  were  the  only  two  of  thirteen 
(15%)  organizations  that  identified  their  metrics  by 
category.  The  tendency  for  organizations  to  ignore  metric 
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classification  seems  to  indicate  a  lack  of  understanding  of 
performance  measures.  Organizations  should  classify  their 
metrics  by  category  type.  Classifying  metrics  into 
definable  categories  would  assist  managers  in  establishing 
effective  measures  that  are  focused  on  key  business 
drivers. 

Furthermore,  the  literature  review  revealed  that 
focusing  performance  measures  on  one  class  or  category  is 
not  prudent.  Metric  classification  would  enable  managers 
to  stratify  their  organization's  performance  measures  to 
ensure  a  balanced  approach  covering  several  categories .  The 
researcher  believes  that  organizations  should  develop  their 
own  categories  as  appropriate,  depending  on  their  specific 
needs.  Classifying  metrics  would  also  enhance  an 
organization's  understanding  of  its  critical  dimensions  and 
impo  r  t an  t  ou  tpu  t  s . 

The  absence  of  standardization  and  metric 
classification  in  the  first  survey  presented  a  challenge 
when  applying  selection  criteria  to  identify  the  25  metrics 
most  relevant  to  OTs.  The  next  subsection  will  classify  the 
survey  results  by  metric  category  to  facilitate  further 
analysis.  As  previously  described,  the  intent  of  this 
study  was  to  obtain  current  information  on  standard 
contract  management  metrics  and  to  determine  the  extent  to 
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which  they  would  be  appropriate  to  manage  Section  845  OTs. 
Based  on  the  literature  review  and  survey  size  constraints, 
25  metrics  from  the  first  survey  were  subjectively 
determined  to  be  most  relevant  to  measuring  OTs.  Those 
served  as  a  base  population  for  the  study's  second  survey. 

c)  Classifying  the  survey  results 

Only  55  of  241  (23%)  metrics  were  identified  by 
category.  To  methodically  select  25  metrics  representative 
of  a  balanced  approach,  covering  several  performance 
categories,  required  classifying  the  remaining  metrics 
according  to  TRADE'S  classification  scheme  outlined  in 
Chapter  III.  While  the  differences  between  performance 
categories  were  clear,  they  were  not  always  obvious  and 
required  intuitive  interpretation.  Classification  results, 
as  depicted  in  Figure  4.3,  reveal:  29  metrics  (12%)  are 
used  to  measure  the  effectiveness  of  contracting 
organizations;  47  metrics  (20%)  are  used  to  measure 
organizational  efficiency;  32  metrics  (13%)  are  used  to 
measure  quality  of  contracts  and/or  contract  organizations; 
43  metrics  (18%)  are  used  to  measure  the  timeliness  of 
various  contract  processes;  51  metrics  (22%)  are  used  to 
measure  organizational  productivity;  and  39  metrics  (17%) 
are  used  to  measure  financial  concerns  of  organizations. 
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Figure  4.3.  Categories  of  Measurement.  [Source:  Developed  by 
Researcher] 


The  research  identified  241  contract  management 
metrics  relatively  evenly  distributed  across  all  six 
categories  (Figure  4.3).  This  distribution  did  not  hold 
true  when  categories  were  stratified  by  organization  (Table 
4 . 1  and  4.2). 
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Results  of  the  suggested  classification,  by 
category  and  organization,  are  summarized  statistically  in 
Tables  4.1  and  4.2.  Table  4.2  depicts  the  percentage 
breakdown  of  classification  categories  by  organization. 
Many  aspects  of  the  buying  organizations  are  revealed  in 
this  breakdown.  For  example,  as  a  post-award  contract 
administration  activity,  DCMC  metrics  emphasize  timeliness 
and  financial  measures.  NAVSEA's  Contracting  Directorate, 
which  provides  contract  support  for  Program  Offices, 
focuses  many  of  its  metrics  on  quality  (e.g.,  customer 
satisfaction,  attitude,  responsiveness,  accountability  of 
contract  professionals,  value  received,  claims  and 
protests).  SPAWAR's  contract  metrics  are  concentrated  on 
measures  of  efficiency.  This  reflects  the  organization's 
interest  in  measuring  the  benefits  of  electronic  commerce 
and  data  exchange.  Appendix  B  lists  contract  management 
metrics  by  classification  category. 

The  classification  of  metrics  was  a  subjective 
process.  Although  the  differences  between  categories  was 
clear,  they  were  not  always  obvious  and  required 
interpretation  to  classify.  Many  categories  were  not 
mutually  exclusive.  However,  for  the  purpose  of  this  study, 
each  metric  was  assigned  to  only  one  category.  Further 
comparison  of  organizations  and  analysis  of  metric 
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categories  are  beyond  the  scope  of  this  thesis,  but  may 
warrant  further  research. 

Classification  results  indicated  that  nine  of 
twelve  organizations  (75%)  measure  the  effectiveness  of 
their  contracts  and/or  contracting  processes  (Figure  4.3). 
Twenty-nine  metrics  were  identified  in  this  category.  Nine 
metrics  measured  the  effectiveness  of  competition  and 
socio-economic  goals.  The  remaining  metrics  captured 
information  on  Contract  Administration  Offices  (CAO) , 
contractor  oversight,  and  product  availability. 


Measures  of  Effectiveness 


Buyng  Command 


Figure  4.4.  Measures  of  Effectiveness.  [Source:  Developed  by 
Researcher] 

Most  of  the  measures  in  this  category  were 
nonjudgmental  and  quantifiable  in  some  way.  For  example, 
NAVICP  monitors  the  effectiveness  of  CAOs  by  measuring  the 
total  number  of  outstanding  UCAs. 


63 


Classification  results  indicated  that  ten  of 


twelve  organizations  (83%)  measured  efficiency.  Forty- 
seven  metrics  were  identified  in  this  category  (Figure 
4.5).  Sixteen  (34%)  measured  staff  workload,  and  23  (49%) 
captured  electronic  commerce  utilization  rates.  Compared 
to  other  organizations,  SPAWAR  measured  the  most  in  this 
category.  Its  metrics  were  oriented  toward  monitoring  the 
benefits  of  electronic  commerce  and  procurement  automation. 


Figure  4.5.  Measures  of  Efficiency.  [Source:  Developed  by  Researcher] 

Classification  results  indicated  that  eight  of 
twelve  organizations  (67%)  measured  the  quality  of  their 
procurement  processes  and  contracting  staff.  Thirty- two 
metrics  were  identified  in  this  category  (Figure  4.6). 
Within  this  category,  15  metrics  (47%)  monitored  customer 
services;  nine  (28%)  measured  protest,  claims,  and 
grievances;  and  three  (9%)  measured  training  utilization 
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rates.  The  remaining  metrics  (16%)  monitored  Procurement 
Management  Review  results,  audit  follow-ups,  and  the 
completeness  of  contractor  alert  lists. 


Figure  4.6.  Measures  of  Quality.  [Source:  Developed  by  Researcher] 

Compared  to  other  organizations,  NAVSEA  dominated 
this  category.  Eight  of  its  metrics  monitored  customer 
service  factors  from  both  a  Program  Office  and  a  contractor 
perspective;  and  four  measured  the  number  of  disputes, 
claims,  protests,  and  contract  cases  in  litigation. 

Classification  results  indicated  that  11  of  12 
organizations  (92%)  measured  the  timeliness  of  their 
procurement  processes.  Forty- three  metrics  were  identified 
in  this  category  (Figure  4.7) .  Twenty- five  (58%)  measured 
procurement  and  negotiations  cycle  time.  For  example,  DLA, 
NAVICP,  NAVSUP ,  ONR,  DOE  and  CAPS  measure  Procurement 
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Figure  4.7.  Measures  of  Timeliness.  [Source:  Developed  by  Researcher] 

Administrative  Lead-Time  (PALT) .  Sixteen  metrics  (37%) 
measured  the  timeliness  of  contract  closeouts,  progress 
payments,  undef initized  contract  actions  (UCA) ,  and 
contract  claims.  None  of  SPAWAR's  21  contract  metrics  were 
classified  in  this  category.  The  majority  of  SPAWAR's 
metrics  combined  attributes  of  both  productivity  and 
timeliness  and,  therefore,  were  better  suited  for 
classification  under  the  category  of  efficiency. 

Classification  results  indicated  that  nine  of 
twelve  organizations  (75%)  measured  productivity.  Fifty- 
one  metrics  were  identified  in  this  category  (Figure  4.7) . 
Based  on  the  literature  review,  46  were  single-dimensional 
measures,  which  represent  basic  and  fundamental  measures  of 
some  process.  Most  metrics  in  this  category  captured 
either  the  numeric  counts  or  dollar  amount  of  contract 
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solicitations,  awards,  and  modifications  issued.  Single¬ 
dimension  measures  of  productivity  are  often  combined  to 
form  multi-dimension  measures  used  in  other  metric 
categories.  Other  measures  captured  the  number  of  active 
suppliers,  purchase  card  users,  receipt  transactions,  and 
work- in-process  by  procurement  types. 


Figure  4.8.  Measures  of  Productivity.  [Source:  Developed  by  Researcher] 

Classification  results  indicated  that  nine  of 
twelve  organizations  (75%)  measured  financial  results. 
Thirty-nine  metrics  were  identified  in  this  category 
(Figure  4.9).  Metrics  from  this  category  varied 
significantly,  comprising  single-dimensional, 
multidimensional  and  judgmental  measures.  Financial 
metrics  focused  on  dollar  values  and  counts,  total 
ownership  costs  (TOC) ,  cost  to  manage,  cost  avoidance,  cost 
savings  and  contract  delinquency.  Many  respondents 
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cautioned  that  financial  metrics  can  be  subjective.  Some 
of  these  measures  are:  contracting  officer  price 
negotiations  savings  and  cost  avoidance;  litigation  cost 
savings  and  avoidance;  process  improvement  cost  savings  and 
avoidance;  and  some  cost  to  manage  metrics. 

Based  on  the  literature  review,  judgmental 
measures  contain  significant  weaknesses.  Information 
collected  usually  involves  weighing  its  value  and  is, 
therefore,  subjected  to  grader  bias.  Managers  should  use 
non judgmental  measures,  whenever  possible,  to  measure 
performance . 


Figure  4.9.  Financial  Measures.  [Source:  Developed  by  Researcher] 


C.  SUMMARY 

This  chapter  summarized  the  results  of  applying 
TRADE'S  metric  classification  scheme  to  the  population  of 
contract  management  metrics  gathered  in  the  first  survey. 
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Prior  to  presenting  this  analysis,  this  chapter  discussed 
the  methodology  of  the  study's  first  and  second  survey,  and 
the  subjective  nature  of  classifying  the  metrics. 

Chapter  V  will  use  a  survey  of  defense  acquisition 
professionals  to  further  investigate  the  extent  to  which 
standard  contract  metrics  are  appropriate  to  measuring  OTs. 
This  chapter  will  first  describe  the  methodology  used  to 
select  the  25  metrics  most  relevant  to  measuring  OTs.  Then 
it  will  present  the  study's  second  survey  and  analyze  the 
results . 
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V.  SURVEY  OF  OTHER  TRANSACTION  METRICS  AND  ANALYSIS 


A.  INTRODUCTION 

Chapter  IV  identified  241  contract  management  metrics 
used  by  nine  Department  of  Defense  (DOD)  organizations,  the 
National  Aeronautics  and  Space  Administration  (NASA) ,  the 
Department  of  Energy  (DOE) ,  and  the  Center  for  Advanced 
Purchasing  Studies  (CAPS) .  The  chapter  summarized  the 
results  of  applying  Training  Resources  and  Data  Exchange's 
(TRADE)  metric  classification  scheme  to  the  population  of 
contract  management  metrics  and  analyzed  the  results . 

Chapter  V  will  show  how  the  researcher  selected  the  25 
metrics  most  relevant  to  measuring  Other  Transactions  (OT) . 
This  subset  of  the  first  survey  served  as  a  basis  for  the 
study's  second  survey.  The  second  survey  had  two 
objectives:  to  determine  the  extent  to  which  metrics  used 
to  manage  standard  contract  performances  are  appropriate  to 
manage  Section  845  OTs;  and  to  determine  to  what  extent  new 
metrics  can  be  developed  that  are  appropriate  for  measuring 
the  use  and  value  of  OTs.  Figure  5.1  illustrates  the 
chapter's  methodology. 

Chapter  V's  analysis  was  conducted  in  two  phases,  each 
presented  in  a  separate  section.  The  first  phase,  presented 
in  Section  B,  is  a  summary  analysis  of  metrics  assigned  to 
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the  six  performance  categories.  This  section  also 
discusses  the  application  of  the  researcher's  metric 
selection  criteria  and  problems  encountered  in  its 
application. 

The  second  phase,  presented  in  Section  C,  will 
describe  the  methodology  used  in  the  second  survey, 
summarize  the  data,  and  analyze  the  responses.  This 
analysis  identified  a  core  set  of  metrics  to  measure  the 
use  and  value  of  Section  845  OTs . 


CH  IV 


r- 
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applicable  to 
R&D  and  OTA 
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Figure  5.1.  Chapter  V  Methodology.  [Source:  Developed  by 
Researcher] 


B*  OT  METRIC  SELECTION  CRITERIA 

1.  Metric  Selection  Criteria 

The  researcher  used  four  criteria  to  select  the 
25  metrics  most  relevant  to  measuring  OTs.  The  first 
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involved  selecting  metrics  that  represented  a  balanced 
approach,  covering  all  six  performance  categories 
(effectiveness,  efficiency,  quality,  timeliness, 
productivity  and  financial) .  TRADE  provides  the  following 
descriptions  of  the  categories.  [Ref.  34:  p.  1-5] 

Effectiveness  indicates  the  degree  to  which  the 
process  output  conforms  to  the  requirement — awards  issued  to 
non- traditional  defense  contractors,  for  example. 

Efficiency  indicates  the  degree  to  which  the  process 
produces  the  required  output  at  a  minimum  resource  cost — 
workload  and  staffing  requirements  for  drafting  and 
negotiating  OTs,  for  example.  Quality  addresses  the  degree 
to  which  customer  requirements  and  expectations  are  met;  an 
example  is  measuring  the  value  received  from  an  OT. 
Timeliness  measures  whether  or  not  a  process  was  done 
correctly  and  on  time,  such  as  measuring  the  cycle  time 
necessary  to  develop  and  insert  a  new  technology. 
Productivity  addresses  the  value  added  by  the  process, 
labor  or  capital  consumed — the  number  of  non-traditional 
contractors  submitting  OT  bids,  for  example.  And  financial 
measures  the  dollar  amount  to  perform  a  task  or  produce  an 
output;  examples  are  monitoring  contractor  cost  share 
ratios  and  measuring  the  cost-to-manage  postaward  contract 
administration . 
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The  second  criterion  selected  metrics  which  were 


best  suited  for  measuring  the  "use  and  value"  of  OTs,  as 
recommended  by  the  DOD  Integrated  Product  Team's  (IPT) 

Final  Report  on  the  Services  Use  of  10  U.S.C  2371  Other 
Transactions  and  845  Prototype  Authorities.  [Ref.  1:  p.  3- 
4]  The  IPT's  report  did  not  explicitly  define  the  terms 
"use"  and  "value."  Based  on  the  legislative  and 
operational  intent  of  OTA  derived  from  the  literature 
review  and  telephone  interviews  with  respondents,  the 
researcher  interpreted  their  meaning  to  be  the  ability  of 
an  OT  to : 

1.  Attract  new,  non- traditional  firms  and 
businesses . 

2 .  Tap  into  commercial  technologies  otherwise 
unavailable  to  DOD. 

3.  Shorten  acquisition  cycle  time. 

4.  Enhance  flexibility  in  the  design  and  development 
process . 

5.  Achieve  affordability  goals. 

The  third  criterion  selected  metrics  that  would 
support  better  management  decision-making,  as  recommended 
by  the  Department  of  Defense  Inspector  General's  (DODIG) 
Audit  Report  98-191  on  the  financial  and  cost  aspects  of 
OTs.  This  report  recommends  establishing  performance 
measures  that  include  the  amount  of  time  required  to  award 
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an  OT  as  opposed  to  a  contract;  the  number  of  new  products 
or  processes  established  as  a  result  of  the  OT;  and  how 
well  Defense  agencies  are  awarding,  managing,  and  competing 
OTs.  [Ref.  6:  p.  16] 

The  fourth  criterion  selected  metrics  directly 
linked  to  the  Assistant  Secretary  of  the  Navy,  Research, 
Development  and  Acquisition  (ASN  [RD&A] )  key  business 
processes  (e.g.,  reductions  in  acquisition  cycle  time  and 
total  ownership  cost).  [Ref.  37:  p.  4]  [Ref.  14] 

2 .  Problems  Encountered 

In  applying  the  selection  criteria  to  the 
contract  metrics,  the  researcher  encountered  two  problems. 
The  first,  based  on  the  literature  review  and  telephone 
interviews  with  respondents,  was  the  difficulty  of  defining 
the  "use  and  value"  of  OTs.  For  the  purposes  of  this 
study,  the  researcher  interpreted  their  meaning  according 
to  the  five  definitions  listed  in  the  subsection  above. 
However,  each  OT  is  unique;  therefore,  its  use  and  value 
may  differ,  depending  upon  a  wide  array  of  goals  specific 
to  each  program's  needs. 

This  can  be  demonstrated  by  the  U.S.  Air  Force's 
use  of  Section  845  OT  on  the  Evolved  Expendable  Launch 
Vehicle  (EELV)  Program.  [Ref.  24:  p.  168]  The  OT  allowed 
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Lockheed  Martin  to  acquire  superior  Russian-made  propulsion 
technology.  The  Russian  technology  was  transferred, 
through  Lockheed  Martin,  to  a  subcontractor  and  then  used 
in  production.  This  gave  the  EELV  program  an  increased 
industrial  base  of  both  domestic  and  foreign  sources  from 
which  to  purchase  advanced  propulsion  system  technology. 

The  OT  enabled  the  EELV  program  to  tap  into  commercial 
technologies  otherwise  unavailable  to  DOD. 

The  second  problem  was  the  necessary  subjectivity 
in  selecting  the  metrics  most  relevant  to  measuring  OTs. 
This  problem  also  cropped  up  when  applying  TRADE'S  metric 
classification  scheme  to  the  body  of  standard  contract 
metrics.  Subjectivity  is  an  inherent  problem  in  the 
selection  and  development  of  OT  metrics,  and  should  be 
taken  into  account  when  considering  the  results  of  this 
study. 


3.  Summary  Analysis  of  Contract  Metrics  Most 
Relevant  to  OTs 

Cumulatively,  ten  percent  of  the  241  contract 
management  metrics  were  subjectively  determined  to  be  most 
relevant  to  measuring  OTs .  Table  5 . 1  and  Appendix  C  list 
the  25  metrics  the  researcher  selected. 
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METRIC  CATEGORY  ACTIVITY  CRITERIA 


Awards  and  Modifications  Issued 

Competition  Goals 

EFFECT 

EFFECT 

AFRL 

NAVSUP 

3 

3 

Active  Suppliers  Per  Purchasing  Employee 

EFFICIENCY 

CAPS 

3 

Workload  And  Staffing 

EFFICIENCY 

NAVSUP 

3 

Reduce  Total  Ownership  Costs  (TOC) :  Development, 
Acquisition,  O&S 

FINANCIAL 

ASN 

4 

Return- On- Investment  (ROI)  [TOC /INVESTMENT 3 

FINANCIAL 

ASN 

4 

Cos t-To- Spend  A  Dollar 

FINANCIAL 

CAPS 

3 

Cost -To -Manage 

FINANCIAL 

NASA 

3 

Solicitations  Issued 

PRODUCT 

AFRL 

2 

Contractors  Assigned  Prime  Contracts 

PRODUCT 

DCMC 

3 

New  Contracts,  Competitive/Non-Competitive 

PRODUCT 

NASA 

3 

New  Financial  Assistance  Instruments  (Grants, 
Cooperative  Agreements) 

PRODUCT 

NASA 

3 

Contract  Awards 

PRODUCT 

ONR 

3 

Customer  Satisfaction 

QUALITY 

NAVSEA 

2 

Value  Received 

QUALITY 

NAVSEA 

2 

Customer  Service 

QUALITY 

NAVSUP 

2 

Reduce  Time  to  Develop  and  Insert  New  Technologies 

TIME 

ASN 

2 

Reduce  Major  Product  Cycle  Time  (Time  to  Develop  & 
Field  Major  System) 

TIME 

ASN 

4 

Shorten  Product  Improvement  Cycle  (Time  to  Develop  & 
Field  Major  Modifications) 

TIME 

ASN 

4 

Purchase  Order  Cycle  Time  (Days) 

TIME 

CAPS 

3 

Negotiation  Cycle  Time 

TIME 

DCMC 

3 

Lead  Time 

TIME 

NASA 

3 

Timeliness 

TIME 

NAVSEA 

3 

Cycle  Time  Reduction 

TIME 

NAVSUP 

4 

Procurement  Administration  Lead  Time  (Contracts) 

TIME 

ONR 

3 

Table  5.1.  25  Contract  Metrics  Most  Relevant  to  OTs.  [Source:  Developed 
by  Researcher] 


According  to  the  literature  review,  focusing 
performance  measures  on  one  category  is  not  prudent . 
Ideally,  metrics  should  be  classified  into  relevant 
categories  that  identify  a  buying  organization's  critical 
dimensions  and  outputs.  Figures  5.2  and  5.3  summarize  the 
results  after  applying  the  first  selection  set  of  criteria 
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to  the  241  contract  metrics.  The  metrics  selected  represent 
a  balanced  approach  that  covers  all  six  performance 
categories  (effectiveness,  efficiency,  quality,  timeliness, 
productivity  and  financial) . 


#  Ilf; 

§  v 

Percent)5 

Survey  Percent 

EFFECTIVENSS 

EFFXCXENCT  £  ^ 

TIMELINESS 

PRODUCT  I VTT  Y 

29 

12% 

2 

8% 

47 

20% 

2 

8% 

32 

13% 

3 

12% 

43 

18% 

9 

36% 

51 

21% 

5 

20% 

39 

16% 

4 

16% 

241 

100% 

25 

100% 

Figure  5.2.  Summary  of  Contract  Metrics  by  Category. 
[Source:  Developed  by  Researcher] 


When  comparing  the  two  surveys  in  Figure  5.2, 
analysis  indicates  a  bias  in  the  measures  of  efficiency  and 
timeliness.  The  researcher  attributes  the  efficiency  bias 
(20%)  to  16  SPAWAR  metrics,  identified  in  the  first  survey, 
which  measure  the  benefits  of  procurement  automation.  If 
removed,  the  results  of  the  first  and  second  surveys  are 
more  closely  aligned  at  13%  and  8%  respectively. 

Figure  5.2  also  indicates  a  bias  toward 
timeliness  measures  (36%) .  The  researcher  attributes  this 
bias  to  a  theme  consistent  throughout  the  literature  review 
and  in  three  out  of  four  of  the  researcher's  selection 
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criteria.  Shortening  acquisition  cycle  time  and  reducing 
the  time  required  to  award  an  OT  were  cited  in  the  DOD 
IPT's  Final  Report,  DODIG  Audit  Report  98-191,  and  ASN 
(RD&A) ' s  1999-2004  Strategic  Plan. 

Figure  5.3  is  a  breakdown  of  the  performance 
categories  and  organizations  from  which  the  25  metrics  were 
selected.  The  Air  Force  Research  Laboratory,  Office  of 
Naval  Research,  Naval  Sea  Systems  Command  and  Defense 
Contract  Management  Command  all  have  experience  drafting 


and/or  administering  OTs . 


Figure  5.3.  Metrics  by  Organization  &  Category.  [Source:  Developed  by 
Researcher ] 


Figure  5.4  summarizes  the  results  of  applying  the 
second,  third,  and  fourth  selection  criteria  to  the  241 
contract  metrics.  Results  reveal  that  five  metrics  (20%) 
were  selected  based  on  the  second  criterion;  15  metrics 
(60%)  were  selected  based  on  the  third  criterion;  and  five 
metrics  (20%)  were  selected  based  on  the  fourth  criterion. 


TOTAL  52324  3  4  2  25 


Figure  5.4.  Metrics  by  Organization  &  Selection  Criteria.  [Source: 
Developed  by  Researcher] 

Based  on  the  criteria  established  by  the  researcher, 

25  standard  contract  metrics  were  selected  to  be  most 
relevant  to  measuring  OTs .  These  metrics  represent  a 
balanced  approach  across  all  performance  categories  and 
were  selected  to  emphasize  recommendations  made  by  both  the 
DOD  IPT  and  DODIG  Audit  Report. 

C.  A  SURVEY  OF  OTHER  TRANSACTIONS 

1 .  Methodology 

The  previous  section  described  how  the  researcher 
subjectively  determined  the  25  metrics  most  relevant  to 
measuring  Other  Transactions  (OT) .  This  subset  of  the 
study's  first  survey  serves  as  a  basis  for  the  second 
survey . 

This  section  will  describe  the  methodology  used  in  the 
second  survey,  present  a  summary  of  data,  and  provide  an 
analysis  of  the  responses.  The  information  in  this  section 
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was  gathered  through  a  survey  of  35  acquisition 
professionals  from  various  DOD  organizations  familiar  with 
OTs.  Twenty  responded — a  57-percent  response  rate.  All 
Government  respondents  were  at  least  at  the  GS-12  level  and 
had  experience  drafting  or  administering  at  least  one  OT. 
Respondents  were  encouraged  to  answer  freely  on  a  non¬ 
attribution  basis. 

The  second  survey  had  two  objectives:  to  determine  the 
extent  to  which  metrics  used  to  manage  standard  contract 
performances  are  appropriate  to  manage  Section  845  OTs;  and 
to  determine  the  extent  to  which  new  metrics  can  be 
developed  that  are  appropriate  for  measuring  the  use  and 
value  of  OTs.  Another  aim  of  the  survey  was  to  determine 
the  extent  to  which  OTs  contribute  to  reductions  in  cycle 
time  and  total  ownership  cost.  Central  to  the  study  is  the 
assumption  that  respondents  are  best  able  to  address  the 
relevance  of  Section  845  OT  metrics,  their  significant 
outputs  and  critical  dimensions.  This  assumption  is  based 
upon  the  fact  that  Section  845  OTA  is  applicable  across  all 
the  DOD  Services . 

The  researcher  based  the  survey's  12  questions  on  the 
literature  review  conducted  in  Chapter  III  and  the  analysis 
presented  in  Chapter  IV.  Respondents  were  encouraged  to 
elaborate  on  any  response.  The  survey  was  not  intended  to 
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be  a  statistically  significant  sampling  of  responses,  but, 
rather,  a  collection  of  opinions. 

2 .  Second  Survey  Results 

The  purpose  of  this  series  of  questions  was  to  survey 
acquisition  professionals  to  determine  how  appropriate  the 
researcher's  25  standard  contract  management  metrics  were 
for  managing  Section  845  OTs.  Respondents  were  provided 
definitions  of  the  metrics  and  rated  each  on  the  basis  of 
their  appropriateness  for  measuring  Section  845  OTs. 
Respondents  selected  from  five  choices:  poor,  fair,  good, 
excellent,  and  not  applicable.  DCMC  respondents,  who  had 
only  post-award  OT  experience,  did  not  rate  several  metrics 
in  question  one.  According  to  COL  Hanson,  Defense  Contract 
Management  Command-Syracuse,  "DCMC  doesn't  get  involved  in 
the  pre-award  action  enough  to  comment  on  the 
appropriateness  of  some  metrics  presented  in  the  survey. " 
Question  number  one  and  the  25  metrics  are  presented  below, 
a.  Responses  to  Question  Number  One 

The  £irst  survey  question  was:  To  what 
extent  are  these  metrics  used  to  manage  standard 
contractual  transactions  appropriate  to  manage  Section  845 
Other  Transactions?  The  25  metrics  and  the  respondents' 
evaluations  of  each  are  shown  below. 
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(1)  Solicitations  issued.  Definition:  The 
quantity  of  solicitations  issued  per  project  or  reporting 
period.  This  metric  is  used  to  track  workload  data  for 
internal  and  external  customers . 


One  of  the  purposes  of  OTA  is  to  encourage 
more  contractor  participation  in  research  and  development 
projects.  Metric  1.1  was  intended  to  solicit  the  value  in 
measuring  the  number  of  OT  solicitations  sent  to  potential 
contractors .  Fifty-eight  percent  of  the  respondents  rated 
this  measure  either  good  or  excellent.  The  remaining 
respondents  (42%)  felt  the  appropriateness  of  this  measure 
was  poor  (34%)  or  fair  (8%) .  Zero  respondents  rated  the 
metric  not  applicable.  Six  DCMC  respondents  elected  not  to 
rate  the  metric . 

The  researcher  expected  more  support  for 
this  measure  and  believes  it  provides  an  indication  of 
competition  achieved.  In  the  researcher's  opinion,  the 
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metric  indicates  the  commercial  sector's  willingness  to 

respond  to  OT  bids,  as  well  as  the  buying  organization's 

propensity  to  seek  out  new  commercial  sources.  According 

to  Diane  Thornwell,  director  of  SPAWAR's  Space  Systems 

Contracting  Division,  "the  acquisition  community  needs  to 

do  a  better  job  at  seeking  out  new  firms  for  OT  business." 

Most  of  the  respondents  who  perceived  OTs  as  valuable  cited 

their  ability  to  attract  new  contractors.  Some 

representative  comments  are  presented  below: 

We  should  be  using  them  [OT]  to  attract  new 
companies  and  new  technologies .  As  well  as 
entering  into  new  partnerships  with  existing 
contractors  that  we  couldn't  get  access  to  by 
using  traditional  FAR  contracts. 

In  my  mind  the  value  of  845  OTs  is  attracting 
non-standard  companies.  Our  focus  should  be  on 
the  success  of  doing  that. 

(2)  Contractors  assigned  prime  contracts. 

Definition :  The  quantity  of  contractors  under  the 
cognizance  of  the  buying  command  who  have  open  prime 
contracts  on-hand  at  the  end  of  the  period.  This  metric  is 
tracked  for  internal  and  external  customers ,  and  for 
workload  indicators . 

Metric  1.2  represents  a  single-dimension 
measure  of  productivity.  This  type  of  metric  is  often 
combined  with  other  single-dimension  measures  and  used  to 
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monitor  effectiveness,  efficiency  and  financial  concerns. 
Seventy- five  percent  of  the  respondents  rated  this  metric 
good  or  excellent.  Twenty-five  percent  of  the  respondents 
rated  the  appropriateness  of  this  metric  as  poor  (8%)  or 
fair  (17%) .  Zero  respondents  rated  this  metric  not 
applicable.  Six  DCMC  respondents  elected  not  to  rate  the 
metric . 


(3)  New  contracts,  competitive/non- 
competitive.  Definition:  The  quantity  of  competitive  and 
non-competitive  contracts  under  the  cognizance  of  the 
buying  command  who  have  open  prime  contracts  on-hand  at  the 
end  of  the  period.  This  metric  is  tracked  for  internal  and 
external  customers,  and  for  competition  indicators. 

Metric  1.3  solicits  the  value  of  comparing 
the  number  of  OT  awards  made  on  a  competitive  versus  non¬ 
competitive  basis.  Fifty-eight  percent  of  the  respondents 
rated  the  metric  good  (17%)  or  excellent  (41%) .  Forty-two 
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percent  rated  the  appropriateness  of  the  metric  fair  (17%) 
poor  (17%)  or  not  applicable  (8%) .  Six  DCMC  respondents 


elected  not  to  rate  the  metric . 


(4)  New  financial  assistance  instruments . 

Definition:  The  quantity  of  new  financial  instruments 
( grants ,  cooperative  agreements ,  or  other  transactions) 
under  the  cognizance  of  the  buying  command  at  the  end  of 
the  period.  This  metric  is  tracked  for  internal  and 
external  customers ,  and  for  workload  indicators . 
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Metric  1 . 4  determines  the  value  of  measuring 
the  number  of  OT  awards.  Sixty-seven  percent  of  the 
respondents  rated  the  measure  as  fair  (8%) ,  good  (25%)  or 
excellent  (34%) .  Zero  respondents  rated  the  metric  poor. 
Other  than  six  DCMC  personnel  who  elected  not  to  rate  the 
measure,  33%  of  the  respondents  felt  the  measure  was  not 
applicable. 


(5)  Contract  actions (new  awards  & 
modifications).  Definition:  The  quantity  of  contract 
actions  (awards  and  modifications )  under  the  cognizance  of 
the  buying  command  at  the  end  of  the  period.  This  metric  is 
tracked  for  internal  and  external  customers ,  and  for 
workload  indicators. 


Metric  1.5  was  intended  to  be  similar  to  the 
previous  question.  Contract  actions  and  new  financial 
instruments  are  essentially  the  same  measures.  Both  are 
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single-dimension  and  capture  information  for  workload  and 
other  metrics.  Eighty  percent  of  the  respondents  rated  the 
measure  as  good  (47%)  or  excellent  (33%) .  Twenty  percent 
of  the  respondents  felt  the  measure  was  poor  (7%)  or  fair 
(13%) .  Zero  respondents  rated  the  metric  not  applicable, 
and  four  DCMC  respondents  elected  not  to  rate  the  metric. 

Comparing  the  responses  to  metrics  1.4  and 

1.5  shows  that  respondents  are  willing  to  measure  the 

number  of  OT  awards.  A  small  percentage  of  respondents  who 

indicated  that  the  metrics  were  poor  or  not  applicable 

generally  felt  that  any  requirement  to  measure  OTs  would 

defeat  the  purpose  of  commercial  streamlining. 

Representative  comments  included: 

Any  requirements  to  measure  defeat  the  purpose  of 
commercial  streamlining.  If  we  start  measuring  it 
usually  means  that  auditors  will  soon  follow  and 
next  thing  you  know  we'll  be  playing  by  some 
other  organization's  rules. 

One  very  basic  concept  of  OTA  is  to  reduce 
burdensome  requirements.  So  by  developing  metrics 
we  impact  that  basic  premise. 

(6)  Contract  awards  (dollars  &  actions) . 

Definition:  The  number  of  contract  awards  by  fiscal  year , 
type,  and  dollar  range. 

Metric  1.6  was  intended  to  see  if 
respondents  rated  contract  awards,  contract  actions  and  new 
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financial  instruments  the  same.  One  hundred  percent  of  the 
respondents  rated  the  measure  as  fair  (29%) ,  good  (29%)  or 
excellent  (42%) .  Zero  respondents  rated  the  metric  not 
applicable  or  poor.  Four  DCMC  respondents  with  post-award 
OT  experience  elected  not  to  rate  the  metric .  Comparing 
the  responses  to  metrics  1.4,  1.5,  and  1.6  confirmed  that 
the  metrics  are  considered  similar  and  appropriate  for 
measuring  OTs . 


(7)  Active  suppliers  per  purchasing 
employee.  Definition:  Total  number  of  purchasing  employees 
engaged  in  award  and/or  administration  per  active 
suppliers . 

Metric  1.7  determined  whether  respondents 
felt  this  workload  indicator  was  relevant  to  OTs.  Seventy- 
two  percent  of  the  respondents  rated  the  metric  either  not 
applicable  (36%) ,  poor  (29%)  or  fair  (7%) .  Twenty-eight 
percent  rated  the  metric  as  good  (14%)  or  excellent  (14%) . 
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Four  DCMC  respondents  elected  not  to  rate  the  metric.  The 


researcher  believes  workload  indicators  can  be  valuable, 
but  agrees  with  the  majority  of  respondents  that  this 
metric's  utility  is  marginal  given  the  relatively  low 
number  of  OTs  being  awarded. 


(8)  Workload  and  staffing.  Definition : 

Ratio  of  the  quantity  of  contract  actions  (awards  and 
modifications )  under  the  cognizance  of  the  buying  command 
at  the  end  of  the  period  to  the  total  number  of  contracting 
employees . 

Metric  1.8  was  also  used  to  determine  the 
relevance  of  workload  measures  to  OTs .  Respondents 
appeared  to  be  more  familiar  with  this  metric  than  with  the 
previous  ones .  Ninety-three  percent  rated  the  metric  as 
fair  (13%),  good  (60%)  or  excellent  (20%).  Zero 
respondents  rated  the  metric  not  applicable,  and  only 
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seven  percent  rated  it  as  poor.  Three  respondents  from  DCMC 

elected  not  to  rate  the  metric.  The  researcher  believes 

that  the  following  comment  made  by  one  respondent  is 

indicative  of  the  tendency  to  accept  this  metric: 

Though  OTs  are  unstructured,  they  require  a  great 
deal  of  time  and  effort  to  draft  and  administer. 
Management  hasn't  yet  decided  how  many  people 
resources  are  needed  to  efficiently  carry  them 
out . 


(9)  Competition.  Definition:  The  extent  of 


competition  achieved. 


One  of  the  purposes  of  OTA  is  to  attract  new 
businesses  and  enhance  competition.  Metric  1.10  solicited 
the  value  in  measuring  the  extent  of  competition  achieved. 
Sixty-two  percent  of  the  respondents  rated  the  measure 
either  good  (31%)  or  excellent  (31%) .  The  remaining 
respondents  (38%)  felt  the  measure  was  poor  (15%) ,  fair 
(15%)  or  not  applicable  (8%) .  Six  respondents  from  DCMC 
elected  not  to  rate  the  metric . 

(10)  Customer  satisfaction.  Definition:  The 
degree  of  customer  satisfaction  in  the  areas  of  timeliness., 
quality,  and  partnering . 


Metric  1.10  determined  the  value  of 
monitoring  OT  customer  service.  The  majority  of 
respondents  indicated  customer  satisfaction  was  a  relevant 
measure.  One  hundred  percent  of  the  respondents  rated  the 
measure  as  fair  (7%),  good  (27%)  or  excellent  (66%).  Zero 
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respondents  rated  the  metric  poor  or  not  applicable,  and 
three  respondents  from  DCMC  elected  not  to  rate  the  metric. 

(11)  Value  received.  Definition:  The  degree 
of  customer  satisfaction  in  value  received  from  contract 
staff. 


Metric  1.11  determined  the  respondents' 
perception  of  measuring  value.  The  literature  review 
reveals  that  value  metrics  are  judgmental.  They  attempt  to 
measure  the  abstract  and  are,  therefore,  subject  to  grader 
bias.  Despite  this  weakness,  seventy-nine  percent  of 
respondents  rated  the  measure  either  good  (36%)  or 
excellent  (43%) .  The  remaining  respondents  (21%)  felt  the 
measure  was  either  fair  (14%)  or  poor  (7%) .  Zero 
respondents  rated  the  metric  not  applicable.  Five 
respondents  from  DCMC  elected  not  to  rate  the  metric. 
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Because  value  measures  are  judgmental,  their 
validity  and  reliability  are  subject  to  question.  In  the 
researcher's  opinion,  this  weakness  can  be  mitigated  when 
both  quantitative  and  qualitative  measures  are  combined 
into  a  single  index  to  better  assess  value. 

(12)  Customer  service.  Definition:  The 
degree  of  customer  satisfaction  with  each  procurement 
organization  in  the  areas  of  timeliness,  quality,  and 
partnering. 


Customer  service  is  also  a  judgmental 
measure.  Comparing  the  responses  to  metrics  1.10  and  1.11 
indicated  that  respondents  support  these  types  of  measures . 
Eighty-six  percent  of  the  respondents  rated  the  measure 
either  good  (59%)  or  excellent  (27%) .  The  remaining 
respondents  (14%)  felt  the  measure  was  either  poor  (7%)  or 
fair  (7%) .  Zero  respondents  rated  the  metric  not 
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applicable,  and  three  respondents  from  DCMC  elected  not  to 
rate  the  metric . 

Although  some  respondents  commented  that 

quantitative  metrics  should  be  developed,  most  preferred  a 

"narrative"  judgmental  approach  instead.  Additional 

comments  from  respondents  are  paraphrased  below: 

Subjective/intuitive  evaluations  can  be  more 
valuable  than  quantitative  metrics  in  many  cases. 

OT  goals  do  not  lend  themselves  to  a  strong  use 
of  metrics .  Quantitative  metrics  are  not  going  to 
tell  the  whole  [OT]  story. 

A  general  statement  of  the  actual  benefit  the  customer 
received  should  be  all  that  is  required. 


(13)  Purchase  order  cycle  time.  Definition: 
Timeliness  of  the  award  process  from  solicitation  to  award. 


The  DODIG  Audit  Report  91-191  recommended 
that  organizations  should  establish  performance  measures 
which  include  the  time  required  to  award  an  OT  as  opposed 
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to  a  contract.  The  following  five  metrics  confirmed  that 
respondents  would  likely  support  this  recommendation. 
Seventy-five  percent  rated  the  measure  as  fair  (8%),  good 
(34%) ,  or  excellent  (33%) .  The  remaining  respondents  rated 
the  metric  poor  (8%)  or  not  applicable  (17%) .  Six 
respondents  from  DCMC  elected  not  to  rate  the  metric. 

(14)  Negotiation  cycle  time.  Definition: 

The  average  quantity  of  days  required  by  the  contract 
administration  office  to  complete  negotiation  during  the 
period.  We  have  this  metric  to  reduce  the  amount  of  time  it 
takes  to  award  a  procurement  action. 


Metric  1 . 14  determined  whether 
respondents  were  concerned  with  the  time  required  to 
negotiate  an  OT.  Ninety- two  percent  of  the 
respondents  rated  the  measure  fair  (17%),  good  (25%) 
or  excellent  (50%) .  Eight  percent  rated  the  metric 


96 


poor.  Zero  respondents  rated  the  metric  not 
applicable,  and  six  respondents  from  DCMC  elected  not 
to  rate  the  metric . 

(15)  Lead  time.  Definition:  Timeliness  of 
the  award  process  from  solicitation  to  award. 


Lead-time  differs  from  negotiation  cycle 
time  in  that  it  captures  the  entire  award  process.  This 
metric  determined  whether  respondents  were  concerned  with 
the  time  required  to  solicit,  negotiate  and  award  an  OT. 
One  hundred  percent  of  the  respondents  rated  the  measure 
fair  (25%) ,  good  (50%)  or  excellent  (25%) .  Zero 
respondents  rated  the  metric  poor  or  not  applicable.  Six 
respondents  from  DCMC  elected  not  to  rate  the  metric. 
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(16)  Timeliness.  Definition:  Timeliness  of 


the  award  process  from  solicitation  to  award. 


Timeliness  and  lead-time  (Metric  1.15)  are 
defined  exactly  the  same,  and  responses  for  both  metrics 
were  consistent.  One  hundred  percent  of  the  respondents 
rated  the  measure  fair  (15%) ,  good  (47%)  or  excellent 
(38%) .  Zero  respondents  rated  the  metric  poor  or  not 
applicable.  Six  respondents  from  DCMC  elected  not  to  rate 
the  metric . 


(17)  Cycle  time  reduction.  Definition: 
Timeliness  of  the  acquisition  process  from  the  beginning  of 
concept  exploration  to  completion  of  prototype. 

Metric  1 . 17  determined  if  respondents  valued 
a  metric  that  measured  cycle  time  reduction  from  beginning 
of  concept  exploration  to  completing  a  prototype.  One 
hundred  percent  of  the  respondents  rated  the  measure  fair 
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(31%) ,  good  (38%)  or  excellent  (31%) .  Zero  respondents 


rated  the  metric  poor  or  not  applicable,  and  five 
respondents  from  DCMC  elected  not  to  rate  the  metric. 


(18)  Procurement  administrative  lead-time 
(PALT).  Definition:  Timeliness  of  the  award  process  from 
solicitation  to  award. 


Metric  1.18  determined  the  value  of 
measuring  Procurement  Administration  Lead  Time  (PALT) . 
Like  the  previous  metrics. "timeliness"  and  "lead  time," 
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PALT  measures  the  timeliness  of  the  award  process.  Ninety- 
two  percent  of  the  respondents  rated  the  measure  fair  (8%), 
good  (59%)  or  excellent  (25%) .  Eight  percent  rated  the 
metric  not  applicable,  and  zero  rated  the  metric  poor.  Six 
respondents  from  DCMC  elected  not  to  rate  the  metric . 

(19)  Time  to  develop  and  field  weapon 
system  modifications.  Definition:  Not  provided. 

I  Tima 

I  43% 


This  metric  determined  whether  respondents 
thought  monitoring  the  time  it  takes  to  develop  and  field  a 
major  weapon  system  modification  was  relevant  to  OTs .  One 
hundred  percent  of  the  respondents  rated  the  measure  fair 
(43%) ,  good  (36%)  or  excellent  (21%) .  Zero  respondents 
rated  the  metric  poor  or  not  applicable.  Four  respondents 
from  DCMC  elected  not  to  rate  the  metric. 
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(20)  Time  to  develop  and  insert  new 


technology.  Definition:  Not  provided. 


This  metric  determined  whether  respondents 
thought  measuring  the  time  it  takes  to  develop  and  insert 
new  technology  was  appropriate.  One  hundred  percent  of  the 
respondents  rated  the  measure  fair  (25%) ,  good  (42%)  or 
excellent  (33%)  .  Zero  respondents  rated  the  metric  poor  or 
not  applicable,  and  six  respondents  from  DCMC  elected  not 
to  rate  the  metric . 

(21)  Time  to  produce  and  field  a  major 
system.  Definition:  Not  provided. 
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This  metric  determined  whether  respondents 
thought  measuring  the  time  it  takes  to  produce  and  field  a 
major  system  was  appropriate.  Eighty- three  percent  of  the 
respondents  rated  the  measure  fair  (25%) ,  good  (25%)  or 
excellent  (33%) .  Zero  respondents  rated  the  metric  poor, 
and  17  percent  rated  it  not  applicable.  Six  respondents 
from  DCMC  elected  not  to  rate  the  metric . 

(22)  Cost -to -manage.  Definition :  Cost  for 
each  procurement  organization  to  manage  procurement 
activity. 

Metric  1.22  solicited  the  value  of  measuring 
costs  associated  with  drafting  and  administering  OTs . 
Eighty-six  percent  of  the  respondents 
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rated  the  metric  either  good  (65%)  or  excellent  (21%) .  T 
remaining  respondents  (14%)  rated  the  metric  fair  (7%)  or 
poor  (7%) .  Five  respondents  from  DCMC  elected  not  to  rate 
the  metric . 

(23)  Cost  to  spend  a  dollar.  Definition: 
Procurement  operating  expense  as  a  percent  of  award 
dollars . 


This  metric  is  similar  to  cost-to-manage . 
Fifty-eight  percent  of  the  respondents  rated  the  metric 
good  (33%)  or  excellent  (25%) .  The  remaining  respondents 
rated  the  metric  fair  (34%)  or  not  applicable  (8%) .  Zero 
respondents  rated  the  metric  poor,  while  six  respondents 
from  DCMC  elected  not  to  rate  the  metric.  Comparing 
metrics  1.22  and  1.23  reveals  that  respondents  gave  a 
combined  rating,  good  and  excellent,  to  cost-to-manage 
(86%)  that  exceeded  cost-to-spend-a-dollar  (58%) . 

(24)  Return  on  investment.  Definition: 
Change  in  Total  Ownership  Cost  divided  by  investment. 


This  metric  determined  whether  respondents 
thought  measuring  Return  on  Investment  (ROI)  was 
appropriate.  Sixty-seven  percent  of  the  respondents  rated 
the  measure  either  good  (42%)  or  excellent  (25%) .  Thirty- 
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three  percent  rated  the  metric  fair,  and  zero  rated  the 
metric  poor  or  not  applicable.  Six  respondents  from  DCMC 
elected  not  to  rate  the  metric. 

(25)  Total  Ownership  cost.  Definition:  Not 

provided. 


This  metric  determined  whether  respondents  thought 
measuring  total  ownership  cost  (TOC)  was  applicable  to  OTs. 
Sixty-two  percent  of  the  respondents  rated  the  measure 
either  good  (31%)  or  excellent  (31%) .  Twenty- three  percent 
rated  the  metric  fair  and  15  percent  not  applicable.  Zero 
respondents  rated  the  metric  poor.  Six  respondents  from 
DCMC  elected  not  to  rate  the  metric . 
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b.  Analysis 


Table  5 . 2  summarizes  the  respondents '  evaluation 
of  the  extent  to  which  25  standard  contract  management 
metrics  are  appropriate  for  measuring  the  use  and  value  of 
Section  845  OTs .  The  table  is  organized  by  measurement 
category.  Metrics  are  highlighted  that  received  a  combined 
quality  rating  greater  than  70  percent,  for  good  and 
excellent.  Individual  metrics  that  received  a  quality 
rating  of  50  percent  or  better,  for  either  good  or 
excellent,  are  blocked. 

Considering  the  responses  summarized  above  led  to 
the  following  conclusions:  (1)  thirteen  metrics  received  a 
combined  quality  rating,  of  good  and  excellent,  that 
exceeded  70  percent;  (2)  the  thirteen  metrics  represent  a 
balanced  approach  across  all  six  measurement  categories; 
and  (3)  customer  satisfaction,  customer  service,  cost-to- 
manage,  and  timeliness  received  the  highest  cumulative 
ratings . 
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METRIC 

CAT 

POOR 

FAIR 

GOOD  EXCEL  N/A 

Awards  and  Modifications  Issued 

EFFECT 

7% 

13% 

o  % 

Competition  Goals 

EFFECT 

15% 

15% 

. 31% . ^31% .  8% 

Active  Suppliers  Per  Purchasing  Employee  EFFIC 

29% 

7% 

14%  14%  36% 

Workload  And  Staffing 

EFFIC 

7% 

13% 

aMmii  o% 

Reduce  Total  Ownership  Costs  (TOC) 

FIN 

0% 

23% 

31%  31%  15% 

Return-On- Investment  (ROI) 

FIN 

0% 

33% 

42%  25%  0% 

Cost-To-Spend  A  Dollar 

FIN 

0% 

33% 

33%  25%  8% 

Cost-To-Manage 

FIN 

7% 

7% 

im^aMpp  o% 

Solicitations  Issued 

PROD 

33% 

8% 

25%  33%  0% 

Contractors  Assigned  Prime  Contracts 

PROD 

8% 

17% 

New  Contracts,  Competitive/Non- 

PROD 

17% 

17% 

17%  42%  8% 

Competitive 

New  Financial  Assistance  Instruments 

PROD 

0% 

8% 

25%  33%  33% 

Contract  Awards 

PROD 

0% 

29%  | 

o% 

Customer  Satisfaction 

Value  Received 

QUAL 

QUAL 

0% 

7% 

7%  | 
14%  ! 

Customer  Service 

QUAL 

7% 

7%  | 

o% 

Time  to  Develop  &  Insert  New  Technology 

TIME 

0% 

25%  | 

°% 

Time  to  Develop  &  Field  Major  System 

TIME 

0% 

25% 

25%  33%  17% 

Time  to  Develop  &  Field  Major  Mods 

TIME 

0% 

43% 

36%  21%  0% 

Purchase  Order  Cycle  Time  (Days) 

TIME 

8% 

8% 

33%  33%  17% 

Negotiation  Cycle  Time 

TIME 

8% 

17%  I 

d3fi8&18SBOi&-  0% 

Lead  Time 

TIME 

0% 

25%  j 

5Q$S 1  25%||  0% 

Timeliness 

TIME 

0% 

15%  | 

0% 

Cycle  Time  Reduction 

TIME 

0% 

31% 

38%  31%  0% 

PALT 

TIME 

0% 

8%  | 

8% 

Table  5.2.  Summary  of  metric  ratings  by  measurement  category.  [Source: 
Developed  by  Researcher] 


The  literature  review  suggested  that  metrics  can 
be  identified  to  help  management  assess  the  value,  and 
control  the  use,  of  Section  845  OTs .  The  respondents 
confirm  this  belief.  Table  5.3  lists,  by  rank,  the 
respondents'  top  13  metrics.  To  qualify,  metrics  had  to 
receive  a  combined  quality  rating,  for  good  and  excellent , 
that  exceeded  70  percent. 

For  thirteen  contract  metrics,  70  percent  of  the 
respondents  believe  they  are  appropriate  for  managing 
Section  845  OTs.  This  fact  is  significant  and  indicates 
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that  both  quantitative  and  qualitative  performance  metrics 
are  applicable  to  OTs . 

Additionally,  the  research  identified  seven 
metrics  having  a  combined  quality  rating  of  80  percent  or 
better.  This  indicates  the  respondents'  willingness  to 
support  customer  satisfaction,  cost-to-manage,  timeliness 
and  workload  performance  metrics.  Analysis  suggests  that 
there  are  two  measurement  categories  considered  most 
relevant  to  OTs:  quality  and  timeliness. 

Meanwhile,  customer  satisfaction  and  negotiation 
cycle  time  received  the  highest  individual  rating  for 


excellent  at  67  percent  and  50  percent  respectively. 
Similarly,  the  share  of  survey  comments  favoring 
intuitive/subjective  measures,  and  raising  concerns 
regarding  the  level  of  effort  required  to  draft  OTs  support 
these  findings. 


METRIC 

CAT 

POOR 

FAIR 

GOOD 

EXCEL 

N/A 

CUM 

RATING 

Customer  Satisfaction 

QUAL 

0% 

7% 

27% 

67% 

0% 

94% 

Customer  Service 

QUAL 

7% 

7% 

60% 

27% 

0% 

87% 

Cos  t -To-Manage 

FIN 

7% 

7% 

64% 

21% 

0% 

85% 

Timeliness 

TIME 

0% 

15% 

iiiii 

38% 

0% 

84% 

PALT 

TIME 

0% 

8% 

Sill! 

8% 

83% 

Workload  And  Staffing 

EFFIC 

7% 

13% 

60% 

20% 

0% 

80% 

Awards  &  Modifications 

Issued 

EFFEC 

7% 

13% 

33% 

0% 

80% 

Value  Received 

QUAL 

7% 

14% 

36%  43% 

0% 

79% 

Negotiation  Cycle  Time 

TIME 

8% 

17% 

25% 

;  50% 

0% 

75% 

Reduce  Time  Insert  New 

Tech 

TIME 

0% 

25% 

42% 

0% 

75% 

Lead  Time 

TIME 

0% 

25% 

50%  : 

25% 

0% 

75% 

Contractors  Assigned  Prime  K 

PROD 

8% 

17% 

42% 

33% 

0% 

75% 

Contract  Awards 

PROD 

0% 

29% 

29% 

0% 

72% 

Table  5.3.  Ranking  of  top  13  metrics  that  received  a  quality  rating 


(good  +  excellent)  above  70  percent.  [Source:  Developed  by  Researcher] 
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c.  Response  to  Questions  Two ,  Three,  and  Four 
Questions  two,  three,  and  four  asked  the 
respondents  to:  Evaluate  the  following  metrics  as  they 
apply  to  Section  845  OT.  Indicate  if  the  metric  is  both 
meaningful,  measurable,  relates  to  your  organization's 
goals,  and  is  economical  to  collect. 

This  series  of  questions  solicited  the 
respondents'  opinions  regarding  OT  metrics  developed  by  the 
researcher.  Respondents  were  provided  the  name  of  the 
metric,  a  description,  and  an  explanation  of  the  process  to 
be  measured.  They  then  answered  four  compliance  questions 
(YES/NO)  and  were  encouraged  to  elaborate  on  any  response. 

(1)  Question  2.  Metric  name :  Section  845 
Prototype  Acquisition  Cycle  Time;  Description :  The  average 
quantity  of  days  required  developing  a  prototype; 
E^lanation  of_  the  process  to  be  measured:  Acquisition 
cycle  time  is  computed  in  days .  The  cycle  time  is 
calculated  by  subtracting  the  Julian  date  BAA  was  issued 
from  the  Julian  date  prototype  was  completed.  Average 
cycle  time  is  calculated  by  adding  the  individual  cycle 
times  in  the  population  and  dividing  by  the  number  of 
Section  845  OTs  in  the  population. 
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YES 

i'"'  'NO 

■WfflSRIWSBBBraiBilBIBi 

79% 

21% 

Relates  to  organization’s  goals 

86% 

14% 

87% 

13% 

Data  is  economical  to  collect 

80% 

20% 

Figure  5.5.  OT  Acquisition  Cycle  Time,  Compliance  Rating. 

[Source:  Developed  by  Researcher] 

Respondents  rated  this  metric 
favorably.  (Figure  5.5).  This  is  consistent  with  the 
results  of  metrics  1.20  (time  to  develop  and  insert  new 
technology)  and  1.21  (time  to  produce  and  field  a  major 
system) .  Many  of  the  respondents  stated  that  the  time 
required  to  develop  research  and  development  prototypes 
would  vary  greatly,  depending  on  the  complexity  of  the 
system  planned  and  the  acquisition  strategy.  Over  time, 
trend  analysis  may  indicate  that  cycle  time  is  reduced. 

(2)  Question  3.  Metric  name :  Attracting  non- 
traditional  defense  contractors;  Description:  The  extent  of 
non- traditional  defense  contractors  attracted  by  Section 
845  Prototype  project;  Explanation  of_  the  process  to  be 
measured:  Volume  of  the  number  of  contractors  submitting 
bids  (defense,  non-defense,  consortia  members) ;  number  of 
non-defense  lower-tier  divisions/contractors  under  prime; 
number  of  prime  and  lower-tier  contractors  that  achieved 
CMRR  status . 
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YES 

MIMI 

88% 

13% 

88% 

13% 

80% 

20% 

collect  1 

67% 

33% 

Figure  5.6.  Attracting  Non- traditional  Defense  Firms, 
Compliance  Ratings.  [Source:  Developed  by  Researcher] 


Respondents  rated  this  metric  favorably. 
(Figure  5.6).  This  is  consistent  with  the  results  of 
metric  1.10  (competition).  Eighty-eight  percent  of 
respondents  indicated  that  this  metric  was  meaningful  and 
related  to  their  organization's  goals.  Eighty  percent  felt 
metric  data  were  measurable;  however,  several  respondents 
commented  that  the  term  "non- traditional"  was  ambiguous  and 
must  be  better  defined  to  accurately  capture  information 
for  this  metric . 

(3)  Question  4.  Metric  name:  OTA  cost 
avoidance;  Des crip t ion:  Costs  avoided  resulting  from  relief 
of  administrative  burdens,  conventional  contract  oversight, 
design/modification  flexibility,  cost  sharing,  etc. 
E^lanation  of_  the  process^  to  be  measured :  Calculate  cost 
avoidance  in  current  year  dollars  over  a  period  not  to 
exceed  the  life  of  the  Section  845  OT.  Cost  avoidance  is 
reported  on  a  one-time  basis;  however,  additional  amounts 
may  be  reported  when  they  are  identified.  Cost  avoidance 
may  be  reported  in  work-hours . 
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YES  ‘ 

•  Meaningful  to  organization 

100% 

0% 

Relates  to  organization’s  goals 

100% 

0% 

31% 

69% 

■  ;;EllH  is  economical  to  collect 

25% 

75% 

Figure  5.7.  OTA  Cost  Avoidance,  Compliance  Rating.  [Source: 
Developed  by  Researcher] 

Respondents  rated  this  metric  favorable 
(Figure  5.7).  This  is  consistent  with  the  results  of 
metrics  1.22  (cost-to-manage)  and  1.23  (cost-to-spend-a- 
dollar) .  One  hundred  percent  of  respondents  indicated  that 
this  metric  was  meaningful  and  related  to  their 
organization's  goals. 

d.  Response  to  Question  Number  Five 
The  fifth  question  was:  From  the  list  below, 
please  identify,  by  ranking  in  order  of  importance,  the 
most  significant  output  of  a  Section  845  "Other 
Transactions . " 

Respondents  were  provided  the  following  six 
selections  to  choose  from: 

1.  Attract  non- traditional  defense  prime  and  sub¬ 
contractors  . 

2 .  Provide  flexibility  in  the  design  and  development 
process . 

3 .  Tap  into  commercial  technology  otherwise  not 
available  to  DOD. 

4.  Reduce  total  ownership  cost. 
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5.  Shorten  acquisition  cycle  time. 

6.  Provide  relief  from  mandates,  statutes,  and 
regulations . 

(1)  Response  and  Analysis.  The  most 
significant  output  of  Section  845  OTs  was  emphasized  in  the 
literature  review  and  is  presented  above.  Analysis 
suggests  there  are  four  outcomes  that  are  most  important  to 
respondents:  the  ability  to  attract  non -traditional  defense 
firms,  flexibility  in  the  design  and  development  process, 
tapping  into  commercial  technology  and  relief  from 
mandates,  statutes  and  regulations.  Specific  trends  are 
evident  in  figures  5.8,  5.9,  and  5.10. 


!Rfl 

1st 

17% 

22%  28%  0% 

0% 

33% 

TOTAL 

17% 

22%  28%  0% 

0% 

33% 

Figure  5.8.  Most  Significant  Outcome;  Respondents  1st  choice.  [Source: 
Developed  by  Researcher] 

Figure  5 . 8  shows  that  respondents  ranked 
relief  from  mandates  statutes  and  regulations  as  the  most 
significant  output  of  an  OT  (33%) .  Tapping  into  commercial 
technology  (28%)  was  ranked  second,  and  flexibility  in  the 
design  and  development  process  was  third  (22%) .  Reducing 
total  ownership  costs  and  shortening  the  acquisition  cycle 
time  ranked  lowest . 
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61% 

6% 

6% 

39% 

Figure 

[Source 

5.9.  Most  Significant  Outcome;  Respondents 
:  Developed  by  Researcher] 

.  1st  and  2 

nd  choices 

When  the  respondents'  first  and  second 
choices  were  analyzed,  the  results  shifted  slightly. 

Figure  5.9  shows  that  respondents  ranked  the  ability  to  tap 
into  commercial  technology  as  the  most  important  output 
(61%) .  Attracting  non-traditional  defense  firms  (45%)  was 
ranked  slightly  above  flexibility  in  the  design  and 
development  process  (44%) .  Reducing  total  ownership  costs 
and  shortening  the  acquisition  cycle  time  were  again  ranked 
lowest . 


:  Attract  ; 
Non- 
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TOC 
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time 
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From 

FAR 

1st 

17% 

22% 

28% 

0% 

0% 

33% 

2nd 

28% 

22% 

33% 

6% 

6% 

6% 

3rd 

17% 

28% 

11% 

6% 

17% 

22% 

TOTAL 

62% 

72% 

72% 

12% 

23% 

61% 

Figure  5.10.  Most  Significant  Outcome;  Respondents  1st'  2nd  and  3rd 
choices.  [Source:  Developed  by  Researcher] 


Figure  5.10  shows  that  respondents  ranked 
the  ability  to  tap  into  commercial  technology  (72%)  and 
flexibility  in  the  design  and  development  process  (72%)  as 
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equally  important  outputs  (tie  for  first)  when  first, 
second  and  third  choices  were  considered.  Attracting  non- 
traditional  defense  firms  (62%)  and  relief  from  mandates 
statues  and  regulations  (61%)  were  ranked  closely  at  third 
and  fourth. 

When  the  first,  second  and  third  choices 
were  analyzed,  the  results  clearly  show  that  reducing  total 
ownership  costs  (TOC)  and  shortening  the  acquisition  cycle 
time  are  considered  the  least  significant  outputs  of  an  OT. 
The  respondents'  low  perception  of  TOC  is  consistent  with 
results  found  in  metric  1.25.  However,  the  respondents' 
low  perception  of  reducing  acquisition  cycle  time 
contradicted  findings  revealed  in  metrics  1.17  (cycle  time 
reduction)  and  1.20  (time  to  develop  and  insert  new 
technology) . 

e.  Response  to  Question  Number  Six 
The  sixth  question  was:  Identify,  by  ranking  in 
order  of  importance,  the  most  critical  dimension  of  a 
Section  845  "Other  Transactions." 

Respondents  were  provided  the  following  five 
selections  to  choose  from: 

1.  Freedom  to  negotiate  intellectual  property  rights 
terms  and  conditions. 
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2 .  Provisions  for  proof  of  concept  and  prototype 
development  prior  to  entering  engineering 
manufacturing  and  design  phase. 

3 .  Minimal  DCAA  involvement . 

4 .  Competing  firms  are  not  required  to  comply  with 
Truth  in  Negotiations  Act  (TINA)  or  Cost 
Accounting  Standards  (CAS) . 

5 .  Provisions  for  cost  and  data  sharing  arrangements 
between  Government  and  contractor. 

(1)  Response  and  Analysis.  According  to  the 
literature  review,  an  organization  must  first  identify  the 
most  significant  output.  Second,  it  must  identify  the 
critical  dimensions  of  performance  for  each  output. 

Critical  dimensions  should  be  derived  from  the  needs  of  the 
customers  who  receive  the  output.  This  question  was 
designed  to  determine  what  the  respondent's  believed  were 
the  most  critical  dimensions  of  a  Section  845  OT. 

The  most  critical  dimension  of  a  Section  845 
OT  was  emphasized  in  the  literature  review.  Analysis 
suggests  that  three  critical  dimensions  are  most  important 
to  respondents :  freedom  to  negotiate  IPR  and  terms  and 
conditions,  proof  of  concept  and  prototype,  cost  and  data 
sharing  arrangements.  Specific  trends  are  evident  in 
figures  5.11,  5.12,  and  5.13. 
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1st 

TOTAL  28%  28%  0%  0%  44% 

Figure  5.11.  Critical  Dimensions  of  an  OT;  Respondents  1st  choice. 
[Source:  Developed  by  Researcher] 

Figure  5.11  shows  that  respondents  ranked 
cost  and  data  sharing  as  the  most  critical  dimension  of  an 
OT  (44%) .  Freedom  to  negotiate  (28%)  and  proof  of 
prototype  (28%)  tied  for  the  second  position.  Minimal  DCAA 
involvement  and  competing  firms  not  having  to  comply  with 
TINA  or  CAS  ranked  lowest. 


TOTAL  61%  50%  0%  33%  55% 

Figure  5.12.  Critical  Dimensions  of  an  OT;  Respondents  1st  and  2nd 
choices.  [Source:  Developed  by  Researcher] 


When  the  respondents'  first  and  second 
choices  were  analyzed,  the  results  shifted.  Figure  5.11 
show  that  respondents  ranked  the  freedom  to  negotiate  as 
the  most  critical  dimension  (61%) .  Cost  and  data  sharing 
(55%)  was  ranked  second,  and  proof  of  prototype  was  third 
(50%) .  Minimal  DCAA  involvement  and  competing  firms  not 
having  to  comply  with  TINA  or  CAS  were  again  ranked  lowest. 
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Figure  5.13.  Critical  Dimensions  of  an  OT;  Respondents  1st'  2nd  and  3rd 
choices.  [Source:  Developed  by  Researcher] 


When  the  respondents'  first,  second  and 
third  choices  were  analyzed,  the  results  clearly  show  the 
least  significant  critical  dimension  to  be  minimal  DCAA 
involvement.  Figure  5.13  show  respondents  ranked  cost  and 
data  sharing  as  the  most  significant  critical  dimension  of 
an  OT  (83%) .  Second  in  importance  was  freedom  to  negotiate 
IPR  and  terms  and  conditions  (78%) .  Proof  of  prototype 
(56%)  and  firms  not  having  to  comply  with  TINA  or  CAS  (55%) 
were  essentially  tied  at  third. 


f .  Responses  to  Question  Number  Seven 
The  seventh  survey  question  was:  To  what  extent 
have  Section  845  "Other  Transactions"  reduced  the  time 
required  to  develop  and  insert  new  technologies? 

This  question  determined  if  respondents  believe 
that  Section  845  OTs  reduce  the  time  required  to  develop 
and  insert  new  technology.  Respondents  were  provided  with 
the  following  six  selections: 
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1.  OTs  significantly  reduce  the  time  required 
(>50%)  . 

2.  OTs  reduce  the  time  required  (25-50%). 

3.  OTs  somewhat  reduce  the  time  required  (10-25%). 

4 .  OTs  do  not  reduce  the  time  required  to  insert  new 
technology. 

5.  OTs  lengthen  the  time  required  to  insert  new 
technology. 

6.  Do  not  know. 


(1)  Response  and  analysis.  Eighty 
percent  of  the  respondents  quantified  their  answer,  while 
20%  did  not.  Twenty  percent  of  the  respondents  felt  that 
Section  845  OTs  contributed  to  at  least  a  50-percent 
reduction  in  the  amount  of  time  required  to  develop  and 
insert  new  technology.  Other  respondents  believed  that 
they  had  contributed  to  a  ten-  to  fifty-percent  reduction 
in  the  time  required  to  develop  and  insert  new  technology. 
Three  DCMC  respondents  elected  not  to  answer  this  question 
for  lack  of  data. 
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fir.  Response  to  Question  Number  Eight 
Question  number  eight  was:  To  what  extent  have  Section  845 
"Other  Transactions"  reduced  acquisition  cycle  time? 

This  question  determined  if  respondents  felt 
Section  845  OTs  reduced  acquisition  cycle  times. 
Respondents  were  provided  the  following  five  selections: 

1.  OTs  significantly  reduced  acquisition  cycle  time 
(>50%)  . 

2.  OTs  reduced  acquisition  cycle  time  (25-50%). 

3.  OTs  somewhat  reduced  acquisition  cycle  time  (10- 
25%)  . 

4.  OTs  have  not  reduced  acquisition  cycle  time. 

5.  OTs  have  lengthened  acquisition  cycle  time. 


(1)  Response  and  Analysis.  Seventy- 
three  percent  of  the  respondents  felt  that  Section  845  OTs 
had  reduced  acquisition  cycle  time.  Seven  percent  of  the 
respondents  indicated  that  OTs  had  increased  acquisition 
cycle  time,  and  13  percent  did  not  know.  Three  DCMC 
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respondents  elected  not  answer  this  question  for  lack  of 
data . 

The  majority  of  the  respondents 
indicated  that  OTs  had  contributed  to  between  a  ten-  and 
twenty-five  percent  reduction  in  time.  The  results 
confirmed  the  DOD  IPT's  preliminary  findings:  the  Services 
are  shortening  the  prototype  cycle  time  [Ref.  1].  However, • 
results  from  question  five  appear  to  imply  respondents 
considered  this  to  be  one  of  the  least  significant  OT 
outputs . 

In  the  researcher's  opinion,  reducing 
acquisition  cycle  time  is  critical  to  both  cost  and 
performance.  Under  the  traditional  approach,  using  a 
standard  FAR/DFAR  contract,  Phases  0  and  I  of  a  major 
defense  acquisition  program  (MDAP)  may  require  in  excess  of 
five  to  nine  years.  As  the  literature  review  revealed. 
Government-imposed  mandates,  statutes  and  regulations  often 
discourage  high-technology  firms  from  bidding  research  and 
development  (R&D)  projects.  Their  primary  concerns  involve 
intellectual  property  rights  (IPR) ,  cumbersome  business 
practices,  socioeconomic  goals,  strict  reporting  standards, 
and  developmental  test  and  evaluation  requirements .  The 
OTA  approach  frees  commercial  firms  from  this  realm  and 
enables  them  to  produce  uninhibited  material  solutions 
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quickly.  This  flexibility  also  enables  contracting 
officers  to  negotiate  IPR  and  address  other  issues;  and  it 
encourages  firms  to  produce  prototypes  more  quickly  than 
the  traditional  approach  allows,  due  in  part  to  the 
incentive  of  taking  the  technology  to  the  commercial 
marketplace.  In  the  researcher's  opinion,  OTA  is  an 
enabler  that  helps  to  shorten  the  early  stages  of  the 
acquisition  cycle. 

h.  Response  to  Question  Number  Nine 
The  ninth  survey  question  was:  To  what  extent 
have  Section  845  "Other  Transactions"  reduced  total 
ownership  cost? 

This  question  determined  if  respondents 
felt  Section  845  OTs  contribute  to  a  reduction  in  total 
ownership  cost  (TOC) .  Respondents  were  provided  the 
following  six  selections: 

1.  OTs  significantly  reduce  total  ownership  cost 
(>50%) . 

2.  OTs  reduce  total  ownership  cost  (25-50%). 

3.  OTs  somewhat  reduce  total  ownership  cost  (10-25%). 

4 .  OTs  do  not  reduce  total  ownership  cost . 

5.  OTs  increase  total  ownership  cost. 
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6. 


Do  not  know. 


(1)  Response  and  Analysis.  Results 
were  mixed.  Forty- two  percent  of  the  respondents  felt  that 
Section  845  OTs  contributed  to  a  reduction  in  TOC.  Forty- 
three  percent  of  the  respondents  indicated  they  did  not 
know  if  OTs  had  decreased  TOC.  Three  DCMC  respondents 
elected  not  to  answer  this  question  for  lack  of  data. 
Although  the  results  provide  some  indication,  they  were 
inconclusive.  Many  respondents  interviewed  by  telephone 
noted  that  several  OTs  were  still  in  effect  and  had  not 
been  completed  or  closed  out.  Results  from  question  five 
show  respondents  considered  TOC  to  be  one  of  the  least 
significant  outputs  of  an  OT. 

i.  Response  to  Question  Number  Ten 
The  tenth  question  was:  To  what  extent  do  you 
believe  Section  845  "Other  Transactions"  WILL  reduce  total 
ownership  cost  over  the  long  run? 
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This  question  determined  if  respondents 
felt  Section  845  OTs  will  eventually  reduce  total  ownership 
cost  (TOC)  .  Respondents  were  provided  the  following  six 
selections : 

1.  OTs  will  significantly  reduce  total  ownership 
cost  (>50%) . 

2.  OTs  will  reduce  total  ownership  cost  (25-50%). 

3  .  OTs  will  somewhat  reduce  total  ownership  cost 
(10-25%). 

4.  OTs  will  not  reduce  total  ownership  cost. 

5.  OTs  will  increase  total  ownership  cost. 

6.  Do  not  know. 


(1)  Response  and  Analysis.  The  majority 
of  respondents,  72  percent,  believe  Section  845  OTs  will 
eventually  reduce  TOC.  Six  percent  of  the  respondent  were 
not  as  optimistic,  while  twenty- two  percent  were  unsure. 
All  respondents  answered  this  question. 
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The  primary  goal  of  many  acquisition  strategies, 
as  articulated  by  the  DODD  5000  series,  is  to  minimize  the 
time  and  cost  required  to  satisfy  an  identified,  validated 
need,  consistent  with  common  sense  and  sound  business 
practices.  According  to  the  Defense  Science  Board's  1996 
Task  Force  on  Defense  Acquisition  Reform,  the  first  two 
phases  of  a  Major  Defense  Acquisition  Program  take  an 
average  of  five  to  nine  years  to  complete.  These  phases, 
which  account  for  three  to  fifteen  percent  of  total  cost, 
are  most  important  because  they  drive  engineering  and 
design,  which,  in  turn,  drives  life  cycle  costs.  When 
properly  employed  as  part  of  the  acquisition  strategy  for  a 
major  acquisition  program.  Section  845  OTs  enable  program 
offices  to  gain  competitive  advantage  in  terms  of 
technology,  design  flexibility,  scheduling  and  cost. 

It  is  difficult  to  measure  reductions  in  total 
ownership  cost  directly  related  to  OTs.  However,  this 
should  not  discourage  acquisition  professionals  from 
striving  to  develop  meaningful  and  accurate  measures  aimed 
at  capturing  this  important  performance  goal . 

j.  Response  to  Question  Number  Eleven 

Question  number  eleven  was:  From  your 
perspective,  should  activities  that  enter  into  Section  845 
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OTs  be  required  to  measure  the  use  and  value  of  those 
instruments? 

The  purpose  of  this  question  was  to 
determine  the  barriers ,  according  to  the  respondents ,  to 
measuring  OTs.  Two  barriers  were  commonly  mentioned: 

1 .  The  perception  that  requirements  to  measure 
defeat  the  purpose  of  commercial  streamlining. 

2.  Collecting  data,  measuring  performance,  and 
reporting  metrics  are  burdensome. 

Representative  comments  about  requirements  to 

measure  the  use  and  value  of  OTs  are  paraphrased  below: 


The  genesis  and  nature  of  OTA  is  to  reduce 
burdensome  requirements  in  an  effort  to 
streamline  the  acquisition  process.  Placing 
metric  burdens  other  than  those  called  for  by  the 
program  manager  and  contracting  officer  reduces 
the  attractiveness  of  such  an  instrument. 

The  value  of  an  OT  is  the  commercial  nature  of 
the  agreement .  Any  requirements  to  measure  defeat 
the  purpose  of  commercial  streaming. 


As  soon  as  you  start  adding  administrative 
requirements  for  measuring  and  collecting 
metrics,  you  cancel  the  benefit  of  the  OTs.  The 
Section  845  was  developed  to  avoid  administrative 
burdens  on  the  government  and  contractor. 

We  have  been  burdened  with  many  requests  for 
reports  and  data  concerning  OTs .  The 
administration  time  required  for  measurement  adds 
to  our  8 -hour  workday. 
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Based  on  the  literature  review,  the 
researcher  anticipated  that  the  respondents  would  be  averse 
to  measuring  OTs .  Research  found  that  sixty  percent  of  the 
respondents  believe  activities,  which  enter  into  Section 
845  OTs,  should  be  required  to  measure  the  use  and  value  of 
those  instruments .  Additional  comments  in  favor  of  OT 
metrics  were: 

Yes,  we  should  be  required  to  measure  OT 
performance.  Section  845  OTs  are  still 
relatively  new  and  not  widely  used.  Therefore, 
this  information  is  important  to  weigh  the  pros 
and  cons  of  using  OTA. 

Yes,  however  we  should  be  cautious  of  metric 
approaches  that  are  overly  structured.  Each 
individual  OT  will  have  it's  own  story  and  will 
probably  not  conform  to  a  standard  metric  format. 

In  my  mind,  the  value  of  Section  845  OT  is 
attracting  non-standard  companies.  Our  focus 
should  be  on  the  success  of  doing  that. 

I  am  in  favor  of  OT  metrics.  If  OTs  don't 
produce  a  value  to  the  organization,  by  opening 
the  competitive  boundaries  under  which 
contractors  compete,  then  why  do  them? 

Yes,  activities  should  be  required  to  measure  OT 
performance.  We  should  be  using  OTs  to  attract 
new  companies  and  new  technologies,  as  well  as  to 
enter  into  new  partnerships  with  existing  DOD 
contractors  that  we  couldn't  get  access  to  by 
using  a  traditional  FAR  contract.  Tracking  this 
type  of  information  is  relevant. 

It's  a  good  idea  to  measure  their  [OT's] 
performance.  However,  it  is  difficult  to  define 
their  use  and  value,  and  hard  to  prove  if  OT  is 
achieving  goals  and  objectives. 
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(1)  Analysis.  The  literature  review 


performed  earlier  in  this  research,  along  with  analysis 
conducted  in  previous  chapters,  indicated  that  the  general 
level  of  support  for  OT  metrics  within  DOD  was  low. 
Approximately  40  percent  of  this  question's  respondents 
confirmed  that  finding.  Their  responses  suggest  two 
barriers  against  OT  metrics:  the  perceptions  that  metric 
requirement  defeats  the  purpose  of  commercial  streamlining 
and  that  metrics  are  a  burden  to  collect. 

The  researcher  believes  the  tendency  for 
these  barriers  to  exist  stems  from  two  sources:  a  lack  of 
understanding  within  DOD  of  performance  metrics,  and  the 
plethora  of  contract  metrics  used  to  measure  the  same 
bottom-line-oriented  performance  that  DOD  has  measured  for 
years . 

In  the  researcher's  opinion,  the 
perception  that  metric  requirements  defeat  the  purpose  of 
commercial  streamlining  indicates  a  lack  of  understanding 
of  both  performance  measures  and  acquisition  reform 
initiatives.  Common  purposes  for  measuring  performance 
include  assessing  management,  evaluating  policy  and  value- 
added  initiatives,  monitoring  risk,  and  determining  whether 
the  organization  is  being  effective  and  efficient.  The 
researcher  believes  that  performance  measures  do  not 
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negatively  impact  commercial  streamlining  reform  efforts. 

In  fact,  if  properly  developed,  they  can  be  used  to  monitor 
the  success  of  reform  initiatives.  For  example,  measuring 
the  success  of  OTA  to  attract  non-traditional  defense  firms 
could  be  used  to^  monitor,  arguably,  one  of  the  authority's 
prime  objectives. 

The  respondents  are  correct;  metrics 
can  be  burdensome  to  collect  and  report .  Survey  results 
presented  in  Chapter  IV  identified  a  sample  of  241  metrics 
used  to  measure  the  performance  of  procurement 
organizations.  DOD  organizations  have  used  this  abundance 
of  contract  metrics  to  measure  the  same  bottom-line- 
oriented  performance  for  years.  Chapter  III  listed  several 
attributes  metrics  should  possess  in  order  to  be  effective. 
Metrics  should  also  be  directly  linked  to  organizational 
output  and  critical  dimensions  of  performance  to  be 
meaningful  and  accepted.  Many  DOD  performance  metrics  lack 
these  attributes  and  are  not  directly  linked  to  key 
business  drivers.  Therefore,  it  is  not  surprising  that 
many  in  the  acquisition  community  perceive  metrics  as 
burdensome . 

However,  a  majority  of  respondents 
indicated  that  establishing  OT  performance  measures  should 
be  required.  This  is  a  significant  finding  and  supports 
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the  need  for  metrics .  Although  most  acquisition 
professionals  support  OT  metrics,  they  favor  a  sensible 
approach — one  that  is  not  overly  structured. 


k.  Response  to  Question  Number  Twelve 
Question  number  twelve  was:  To  what  extent  can 
metrics  be  used  to  provide  insight  into  whether  Section  845 
OTs  are  better,  faster,  and  cheaper  methods  of  conducting 
business? 


This  question  solicited  the 

respondents'  opinions  regarding  the  extent  to  which  OT 

metrics  provide  insight  into  whether  Section  845  OTs  are 

better,  faster,  and  cheaper  than  standard  FAR  type 

contracts.  Representative  comments  are  paraphrased  below: 

Metrics  are  definitely  important  in  determining 
whether  Section  845  OTs  are  a  better,  faster,  and 
cheaper  method  of  conducting  business.  Cost 
performance  and  cycle  time  measures  could  be  used 
to  determine  the  effect  of  OT  on  acquisition 
cycle  and  costs  to  the  government. 


Those  metrics  which  are  easily  collected  and 
reflect  accurately  the  cost  savings,  lead  time 
reduction,  etc.  can  be  used,  but  business  methods 
not  easily  captured  with  metrics  should  not  be 
brought  into  metric  evaluation. 
Subjective/intuitive  evaluations  can  be  more 
valuable  then  metrics  in  many  cases. 

OT  metrics  should  focus  on  cycle  time  and  be  able 
to  assess  their  cost-to-manage  vs.  traditional 
contents . 
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OTs  are  faster  because  they  allow  you  to  work 
problems  outside  the  box  [FAR/DFARS] .  Metrics 
could  be  used  to  monitor  this  information. 


Other,  less  positive,  responses  are  presented 


below: 

Metrics  can  obviously  be  used  to  measure  these 
elements.  However,  extreme  caution  should  be  used 
in  developing  OT  metrics .  One  very  basic  concept 
of  OTs  is  to  reduce  burdensome  requirements.  So, 
by  developing  these  metrics,  we  impact  that  basic 
premise. 

Keep  OT  metrics  light.  Too  much  statistical 
analysis  of  OT  vs.  FAR  contracts  is  probably 
unhealthy  for  both  types . 

OT  metrics  should  be  used  to  assess  the 
successfulness  of  OSD/ASN  reform  initiatives. 

They  [OT  metrics]  should  not  be  used  to  micro 
manage  program  offices. 

Metrics  in  the  OT  environment  make  me  nervous . 

In  general,  metrics  have  a  way  of  mystically 
growing  to  dominate  a  process . 

(1)  Analysis.  The  responses  regarding 
whether  metrics  can  be  used  to  indicate  if  OTs  are  better, 
faster,  and  cheaper  are  mixed.  Many  respondents 
characterize  OT  metrics  as  promising  and  potentially 
useful.  This  characterization  supports  earlier  findings 
that  identified  13  standard  contract  metrics,  which 
received  a  combined  quality  rating  of  70  percent, 
applicable  to  OTs.  This  indicates  that  metrics  are 
appropriate  and  relevant  to  measuring  OTs.  Other 
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respondents  feel  metrics  would  not  yield  meaningful 
information  and  would  burden  the  acquisition  workforce. 

3.  A  Core  Set  of  Metrics  Applicable  to  Section  845 
OTs 

The  study's  second  survey  had  two  objectives. 
First,  to  determine  the  extent  to  which  metrics  used  to 
manage  standard  contract  performances  are  appropriate  to 
manage  Section  845  OTs.  Table  5.2  above  summarized  the 
research  findings  applicable  to  this  question.  The  second 
objective  was  to  determine  if  new  metrics  can  be  developed 
which  are  appropriate  for  measuring  the  use  and  value  of 
OTs . 

The  researcher  recommends  four  possible  measures  that 
may  serve  as  a  core  set  of  metrics  applicable  to  Section 
845  OTs.  They  are,  attracting  non- traditional  defense 
firms,  prototype  acquisition  cycle  time,  customer 
satisfaction  and  timeliness.  Definitions  of  the  metrics 
are  provided  in  questions  3,  2,  1.9  and  1.17  respectively. 

The  researcher  considers  the  first  two  metrics, 
attracting  non-traditional  defense  firms,  and  prototype 
acquisition  cycle  time,  as  organizational  measures.  These 
measures  are  aimed  at  upper  echelon  organizations,  such  as 
ASN  (RD&A) .  These  metrics  capture  information  directly 
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relevant  to  ASN  (RD&A)'s  key  business  process  goals  for 
reducing  acquisition  cycle  time  and  enhancing  the 
integration  of  civilian  and  military  buying  organizations. 
Metric  information  could  be  collected  by  either  the  DD  Form 
2759  Report  of  Other  Transactions  for  Prototype  Projects  or 
it's  Data  Element  Narrative. 

Although  reducing  prototype  acquisition  cycle  time  was 
not  considered  a  significant  outcome  when  compared  to 
others,  86  percent  of  respondents  felt  the  measure  related 
to  their  organization's  goals,  87  percent  indicated  the 
metric  data  was  measurable  and  80  percent  felt  it  was 
economical  to  collect.  In  fact,  a  similar  metric  "reducing 
time  required  to  insert  new  technology"  received  a  combined 
quality  rating  of  75  percent  and  was  ranked  ninth  by 
respondents.  According  to  the  literature  review  conducted 
in  Chapter  III,  this  metric  possesses  the  attributes  of  a 
successful  measure  and  appears  to  be  a  useful  performance 
indicator  for  one  of  the  prime  objectives  of  OTA-  the 
ability  to  insert  and  develop  new  technology  into  a 
deliverable  prototype. 

The  researcher  also  recommends  attracting  non- 
traditional  defense  firms  as  a  core  OT  metric.  Respondents 
ranked  attracting  non-traditional  defense  firms  as  the 
second  most  significant  output  of  an  OT,  behind  the  ability 
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to  tap  into  commercial  technology  otherwise  unavailable  to 
DOD.  Eighty-eight  percent  of  respondents'  felt  the  measure 
was  meaningful  and  indicated  it  related  to  their 
organization's  goals,  80  percent  claimed  the  metric  data 
was  measurable  and  67  percent  felt  it  was  economical  to 
collect.  Although  the  metric  possesses  the  attributes  of  a 
successful  measure,  the  term  non-traditional  defense  firm 
would  need  to  be  defined  in  order  to  make  the  metric  both 
valid  and  reliable. 

While  the  previous  metrics  are  organizational 
measures,  that  serve  the  needs  of  the  upper  management 
echelon,  the  researcher  also  recommends  two  process  metrics 
applicable  to  buying  organizations:  customer  satisfaction 
and  timeliness. 

The  most  highly  rated  standard  contract  metric 
applicable  to  OTs  was  customer  satisfaction.  This  metric 
received  a  combined  quality  rating  of  94  percent  and  was 
ranked  first  by  respondents.  Respondents  were  not  given 
the  choice  to  select  customer  service  as  a  significant 
outcome.  However,  they  rated  the  flexibility  OT  provides 
the  program  office  in  the  design  and  development  process, 
as  the  most  significant  OT  output;  along  with  OTs  ability 
to  tap  into  commercial  technology  otherwise  unavailable. 

In  the  researcher's  opinion,  both  these  outputs  are 
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directly  relevant  to  a  program  office's  perception  of 
customer  satisfaction. 

Additionally,  this  metric  is  closely  aligned  with  the 
respondent's  evaluation  of  the  critical  OT  dimensions. 
Specifically,  the  ability  to  share  data  and  negotiate  key 
terms  and  conditions.  Although  customer  satisfaction  is  a 
judgmental  measure  and  subject  to  bias,  many  respondents 
indicated  willingness  to  report  intuitive  style 
evaluations,  rather  than  quantitative  measures. 

Another  core  metric  the  researcher  recommends 
monitoring  is  the  timeliness  of  the  OT  procurement  process. 
This  metric  was  ranked  fourth  by  respondents  and  received  a 
combined  quality  rating  of  84  percent.  Table  5.14  lists 
the  researcher's  core  set  of  metrics  applicable  to  Section 
845  OTs. 


:  se  ION  345  ot  Measurement  Quality 

Attracting  Non-Traditional  Defense  Firms 
Section  845  Prototype  Acquisition  Cycle  Time 
Timeliness  of  OT  Award  Process 

Customer  Satisfaction 

EFFECTIVENESS 

TIMELINESS 

TIMELINESS 

QUALITY 

NA 

NA 

84% 

94% 

Table  5.14.  Core  OT  Metrics.  [Source:  Developed  by  Researcher] 


D.  SUMMARY 

This  chapter  provided  a  summary  analysis  of  the  241 
standard  contract  metrics  assigned  to  the  six  performance 
categories.  The  first  part  of  the  analysis  described  the 


135 


researcher's  metric  selection  criteria  and  problems 
encountered  in  its  application.  Next,  the  researcher 
analyzed  the  25  metrics  subjectively  determined  to  be  most 
relevant  to  measuring  OTs .  The  results  were  then  used  as  a 
basis  for  the  study's  second  survey. 

Part  two  of  the  chapter  presented  the  methodology  used 
in  the  second  survey  and  provided  an  analysis  of  the 
results .  This  analysis  determined  the  extent  to  which 
standard  contract  metrics  are  appropriate  for  managing 
Section  845  OTs,  and  identified  a  core  set  of  metrics  to 
measure  their  use  and  value . 
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VI.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  INTRODUCTION 

This  chapter  will  present  conclusions  and 
recommendations  based  on  the  research,  answer  the  primary 
and  subsidiary  research  questions,  and  suggest  areas  for 
further  research. 

B.  CONCLUSIONS 

Several  conclusions  can  be  drawn  from  the  research 
conducted  in  this  study.  First,  survey  one  identified  241 
contract  management  metrics  used  to  measure  the  performance 
of  12  procurement  organizations.  Results  indicate  that 
organizations  measure  similar  procurement  processes; 
however,  fewer  than  ten  percent  of  the  metrics  identified 
shared  the  same  title.  Additionally,  respondents  confirmed 
that  their  organizations  were  not  using  metrics  to  measure 
the  performance  of  OTs . 

Second,  the  majority  of  respondents  (70%)  were  unable 
to  provide  a  formal  metric  [contract]  listing.  As 
explained  in  Chapter  IV,  Organizations  were  expected  to 
provide  a  metric  listing  that  identified  metrics  by 
measurement  category,  title,  definition,  objective,  data 
source,  computation  method  and  frequency  of  measure.  The 


137 


absence  of  standardization  and  metric  classification 
presented  a  challenge  when  identifying  the  metrics  most 
relevant  to  OTs .  By  applying  Training  Resources  and  Data 
Exchange's  (TRADE)  classification  scheme  to  the  base 
population  of  contract  metrics,  the  researcher  was  able  to 
organize  metrics  by  measurement  category  (effectiveness, 
efficiency,  quality,  timeliness,  productivity  and 
financial).  TRADE'S  classification  scheme,  along  with  the 
metric  selection  criteria  detailed  in  Chapter  V,  enabled 
the  researcher  to  subjectively  determine  the  25  metrics 
most  relevant  to  OTs . 

Third,  of  the  25  standard  contract  metrics 
subjectively  determined  to  be  most  relevant  to  OTs,  seven 
received  a  combined  quality  rating,  for  good  and  excellent, 
which  exceeded  80  percent.  Chapter  V  showed  that  the 
metrics  considered  the  most  appropriate  for  measuring  OTs 
are  customer  satisfaction,  customer  service,  cost-to-manage 
and  timeliness. 

Fourth,  as  detailed  in  Chapter  V,  the  most  significant 
output  of  a  Section  845  OT  was  blurred  at  the  first,  second 
and  third  levels.  Respondents  considered  Section  845  OT's 
most  significant  outputs  to  be  its  ability  to  tap  into 
commercial  technology,  to  attract  non-traditional  defense 
firms  and  to  provide  flexibility  in  the  design/development 


process.  As  Figure  5.9  shows,  respondents  judged  the  least 
significant  output  of  an  OT  to  be  its  ability  to  reduce 
total  ownership  costs  (TOC) .  Respondents  also  indicated 
that  OT's  most  critical  dimensions  are  its  cost  and  data 
sharing  attributes. 

Fifth,  the  majority  of  respondents  felt  that  OTs 
contributed  to  at  least  a  ten-percent  reduction  in  the  time 
required  to  develop  and  insert  new  technology,  and  to  a 
significant  reduction  in  acquisition  cycle  time. 

Sixth,  a  slight  majority  of  respondents  (60%)  favored 
establishing  Section  845  OT  performance  measures;  however, 
most  were  against  taking  an  overly  structured  approach. 

As  Chapter  V  explained,  many  respondents  characterized  OT 
metrics  as  promising  and  potentially  useful;  however, 
results  were  mixed  as  to  whether  organizations  should  be 
required  to  measure  their  use  and  value. 


C.  RECOMMENDATIONS 

This  study  has  provided  information  about  standard 
contract  performance  metrics  to  determine  the  extent  to 
which  metrics  are  appropriate  for  managing  OTs.  Several 
recommendations  draw  on  this  information. 

First,  the  acquisition  and  contracting  community  must 
be  encouraged  to  measure  performance.  This  would  support 
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better  management  decision-making,  thus  improving  outcomes. 
Organizations  should  concentrate  metrics  on  key  performance 
drivers  that  are  meaningful,  that  relate  to  the 
organization's  goals,  and  that  are  simple  to  understand, 
clearly  defined  and  economical  to  collect.  Ideally,  the 
maximum  number  of  metrics  any  organization  should  use  is 
20. 

Organizations  should  maintain  a  command  list  of 
contract  metrics  complete  with  title,  definition, 
classification,  measurement  objective,  data  source, 
computation  methodology,  and  frequency  of  measure.  This 
practice  would  assist  acquisition  professionals  in 
establishing  effective  metrics  focused  on  key  business 
drivers.  It  also  would  enhance  an  organization's 
understanding  of  its  critical  dimensions  and  important 
outputs . 

The  second  recommendation  is  that  organizations  should 
place  a  greater  emphasis  on  developing  Section  845  OT 
metrics.  As  in  the  Department  of  Defense  (DOD)  Integrated 
Product  Team's  preliminary  findings  and  recommendations 
from  the  Office  of  the  Inspector  General,  DOD  (DODIG)  Audit 
Report,  activities  that  enter  into  Section  845  OTs  should 
develop  and  employ  metrics  that  measure  the  use  and  value 
of  OTs.  Survey  results  suggest  that  respondents  are 
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willing  to  support  standard  contract  metrics,  such  as 
customer  satisfaction,  workload  performance,  cost-to-manage 
and  timeliness  of  the  award  process.  The  researcher 
believes  that  Section  845  OT  metrics  should  focus  on 
monitoring  organizational  goals  at  the  Secretariat  level . 
Two  of  these  goals,  reducing  the  time  required  to  insert 
new  technology  and  attracting  non- traditional  defense  firms 
to  participate  in  DOD  R&D  projects,  are  the  primary 
objectives  of  other  transaction  authority  (OTA) .  The 
survey  respondents  appeared  willing  to  support 
organizational  metrics  aimed  at  monitoring  acquisition 
cycle  time  and  attracting  non- traditional  defense  firms. 

Finally,  the  researcher  recommends  establishing  four 
measures  to  serve  as  a  core  set  of  metrics  applicable  to 
all  Section  845  OTs :  attracting  non- traditional  defense 
firms;  monitoring  prototype  acquisition  cycle  time; 
customer  satisfaction;  and  timeliness  of  the  award  process. 
Such  metrics  would  provide  information  that  could  lead  to  a 
better  understanding  of  the  use  and  value  of  Section  845 
OTs . 
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D.  RESEARCH  QUESTIONS 

This  section  provides  answers  to  the  research 
questions  posed  in  Chapter  I .  The  primary  research 
question  this  thesis  sought  to  answer  was: 

1.  To  what  extent  are  metrics  used  to  measure 
standard  contractual  transactions  appropriate  to  manage 
Section  845  "Other  Transactions?" 

The  study  suggests  that  13  of  241  metrics  (5%)  used  to 
manage  standard  contractual  transactions  are  appropriate 
for  managing  Section  845  OTs.  The  data  analysis,  presented 
in  Chapter  V,  identifies  the  metrics  by  title,  performance 
category  and  level  of  acceptance. 

Subsidiary  research  questions  included: 

2.  What  are  Other  Transactions,  and  how  are  they 
distinguished  from  standard  contractual  transactions? 

In  the  context  of  the  Federal  Acquisition  Regulations 
(FAR) ,  OTs  are  not  contracts,  cooperative  agreements  or 
grants.  Rather,  they  are  funding  instruments  between 
Government  and  commercial  firms  or  consortiums.  OTs  are 
used  to  carry  out  basic,  applied,  advanced  research  and 
prototype  projects.  OTs  are  different  from  standard 
contractual  transactions  because  they  are  exempt  from  many 
mandates,  statutes  and  regulations;  this  provides 
negotiators  the  flexibility  to  craft  commercial -like 
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agreements  that  can  address  a  broad  spectrum  of  issues 
specific  to  individual  program  needs. 

3.  What  metrics  are  commonly  used  to  manage  standard 
contractual  transactions? 

The  summary  analysis  of  the  first  survey,  presented  in 
Chapter  IV,  identified  241  standard  contract  metrics  used 
by  12  buying  organizations  (Figure  4.2) .  Appendix  A  lists 
contract  metrics  by  organization. 

The  absence  of  standardization  and  metric 
classification  in  the  first  survey  prompted  the  researcher 
to  classify  the  metrics  into  six  performance  measurement 
categories.  These  categories  were  effectiveness, 
efficiency,  quality,  timeliness,  productivity  and 
financial.  Analysis  of  the  classification  results 
indicated  the  metrics  were  fairly  evenly  distributed  across 
all  six  measurement  categories  (Figure  4.3) . 

4.  To  what  extent  are  these  metrics  suitable  for 
managing  Other  Transactions  and  Section  845  Prototype 
Transactions? 

Based  on  criteria  established  in  this  research,  25  of 
241  standard  contract  metrics  were  subjectively  determined 
to  be  suitable  for  managing  Section  845  OTs.  Furthermore, 
the  study  showed  that  respondents  rated  13  of  25  metrics  to 
be  most  appropriate  for  managing  Section  845  OTs.  Table 
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5.3  lists,  by  rank,  the  respondents'  top  13  metrics.  To 
qualify,  metrics  had  to  receive  a  combined  quality  rating, 
for  good  and  excellent,  that  exceeded  70  percent.  The  top 
13  metrics  are  both  qualitative  and  quantitative  measures 
representing  all  six  measurement  categories. 

5.  Can  a  core  set  of  metrics,  common  to  all  Section 
845  Other  Transactions,  be  developed? 

The  study  suggested  that  four  performance  measures 
could  serve  as  a  core  set  of  metrics  common  to  all  Section 
845  OTs:  attracting  non-traditional  defense  firms; 
monitoring  prototype  acquisition  cycle  time;  customer 
satisfaction;  and  timeliness  of  the  award  process.  This 
suggestion  was  based  on  analysis  presented  in  Chapter  V. 

6 .  What  are  the  advantages  of  developing  metrics  for 
managing  Section  845  Other  Transactions  and  what  are  the 
barriers  to  implementing  them? 

This  study  suggested  that  metrics  can  assist 
management  in  assessing  the  value,  and  controlling  the  use, 
of  Section  845  OTs.  First,  metrics  could  evaluate  whether 
OTs  are  meeting  value-added  objectives  such  as  attracting 
non-traditional  defense  firms,  tapping  into  commercial 
technology  or  reducing  prototype  acquisition  cycle  time. 
Second,  buying  organizations  could  implement  timeliness  and 
customer  satisfaction  metrics  to  monitor  how  well  the  award 
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process  was  functioning.  This  could  also  identify  OT 
performance  disparities  that  should  be  analyzed  and 
eliminated. 

The  study  suggested  that  two  barriers  exist  when 
implementing  OT  metrics .  They  are  the  perception  by  some 
that  measurement  requirements  defeat  the  purpose  of 
commercial  streamlining  and  that  metrics  are  a  burden  to 
collect.  The  researcher  suggested  in  Chapter  V  that  the 
tendency  for  these  barriers  to  exist  stems  from  two 
sources:  a  lack  of  understanding  within  DOD  of  performance 
metrics,  and  the  plethora  of  meaningless  contract  metrics 
used  to  measure  the  same  bottom-line  performance  that  DOD 
has  measured  for  the  past  twenty  years . 

The  implementation  of  OT  performance  measures  should 
involve  as  many  cognizant  employees  as  possible  to 
stimulate  ideas  and  reinforce  the  notion  that  performance 
measurement  is  a  team  effort  requiring  buy-in  from  all 
involved  in  order  to  succeed. 

E.  RECOMMENDATIONS  FOR  FURTHER  RESEARCH 

This  study  focused  on  developing  Section  845  OT 
metrics.  The  researcher  has  identified  three  areas  in 
which  further  research  would  be  useful.  The  first 
recommendation  is  to  conduct  a  comparative  analysis  of 
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buying  organizations'  contract  metrics.  Are  organizational 
goals  and  objectives  linked  to  contract  performance 
metrics? 

This  study  revealed,  intuitively,  that  OTs  have  been 
successful  in  attracting  non-traditional  sources  of  supply, 
and  allowing  DOD  to  tap  into  commercial  technology 
otherwise  unavailable.  How  successful  have  OTs  been  in 
achieving  these  objectives?  One  survey  respondent  stated 
that  DOD  doesn't  seek  out  new  non-traditional  firms  as 
aggressively  as  it  should.  Therefore,  the  second 
recommendation  is  to  conduct  a  study  to  quantitatively 
determine  the  extent  to  which  OTs  have  attracted  non- 
traditional  defense  prime  and  subcontractor  firms. 

This  study  suggested  that  buying  organizations  could 
submit  metric  data  via  the  DD  Form  2759  and  its 
accompanying  data  element  narrative.  The  third 
recommendation  is  to  study  how  OT  metric  data  could  be 
gathered.  Would  metric  data  be  precise,  valid  and 
accurate,  or  a  burden  to  collect? 
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APPENDIX  A.  CONTRACT  MANAGEMENT  PERFORMANCE  METRICS 


Metric _ . _ Organization 

Funding  Documents  Received  AFRL 

Awards  &  Modifications  Issued  AFRL 

Purchase  Requests  Transmitted  AFRL 

Solicitations  Issued  AFRL 

Solicitations  Transmitted  to  ACO  AFRL 

Solicitations  Transmitted  to  Contractors  AFRL 

Purchase  Requests  Transmitted  to  DFAS  AFRL 

Purchase  Requests  Transmitted  to  Files /Archives  AFRL 

Purchase  Requests  Transmitted  to  Originator  AFRL 

Reduce  Total  Ownership  Costs  (EMD,  PRODUCTION,  O&S)  ASN 
Return  on  Investment  (ROI)  [TOC /INVESTMENT]  ASN 

Reduce  Major  Product  Cycle  Time  ASN 

(Time  to  Develop  and  Field  System) 

Reduce  Time  to  Develop  and  Insert  New  Technologies  ASN 
Shorten  Product  Improvement  Cycle  ASN 

(Time  to  Develop  and  Field  Major  Modifications) 

Active  Suppliers  per  Professional  Purchasing  CAPS 

Employee 

Active  Suppliers  per  Purchasing  Employee  '  CAPS 

Percent  of  Purchase  Transactions  Processed  through  CAPS 
Electronic  Commerce 

Percent  of  Purchase  Transactions  Processed  via  CAPS 

Procurement  Card 

Percent  of  Services  Purchases  handled  by  the  CAPS 

Purchasing  Department 

Percent  of  Total  Purchases  handled  by  the  Purchasing  CAPS 
Department 

Purchase  Dollars  per  Professional  Purchasing  CAPS 

Employee 

Purchase  Dollars  per  Purchasing  Employee  CAPS 

Purchasing  Employees  as  a  Percent  of  Company  CAPS 

Employees 

Cost  To  Spend  A  Dollar  CAPS 

Percent  of  Purchase  Dollars  Spent  with  Minority-  CAPS 

owned  Suppliers 

Percent  of  Purchase  Dollars  Spent  with  Women-owned  CAPS 
Suppliers 

Purchase  Dollars  as  a  Percent  of  Sales  Dollars  CAPS 

Purchase  Dollars  Spent  per  Active  Supplier  CAPS 

Purchase  Operating  Expense  Dollars  as  a  Percent  of  CAPS 
Sales  Dollars 

Purchasing  Operating  Expense  Dollars  per  Active  CAPS 
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Supplier 

Change  in  Number  of  Active  Suppliers  During  the  CAPS 

Reporting  Period 

Percent  of  Active  Suppliers  Accounting  for  90  CAPS 

Percent  of  Purchase  Dollars 

Sales  Dollar  per  Purchasing  Employee  CAPS 

Average  Annual  Training  Hours  per  Professional  CAPS 

Purchasing  Employee 

Purchase  Order  Cycle  Time  (days)  CAPS 

M/C  Request  for  Waivers /Request  For  Deviations  per  DCMC 
1,000  Contracts 

Major  Software  Recommendations  Adopted  DCMC 

Corrective  Action  Request  Cost  Avoidance  DCMC 

Quantity  of  Overage  Contracts  for  Closure  DCMC 

Unliquidated  Amount  of  Prime  Contracts  On-Hand  DCMC 

Amount  of  Loss,  Damage  and  Destruction  (LDD)  DCMC 

Contracting  Officer  Price  Negotiations  Savings  and  DCMC 
Avoidance 

Cost  Overruns  on  Major  Programs  DCMC 

Litigation  Cost  Savings  and  Avoidance  DCMC  . 

Percent  of  Excess  Property  Reutilized  and  Sales  DCMC 

Proceeds 

Process  Improvement  Cost  Savings  and  Avoidance  DCMC 

Quantity  of  Price  Negotiations  DCMC 

(Used  to  monitor  cost  of  negotiations) 

Contractors  Assigned  Prime  Contracts  DCMC 

Obligated  Amount  of  Prime  Contracts  On-Hand  DCMC 

Prime  Contracts  On-Hand  DCMC 

Completeness  of  the  Contractor  Alert  List  (CAL)  DCMC 

Customer  Satisfaction  Rating  DCMC 

Grievances  DCMC 

Contract  Closeout  Cycle  Time  DCMC 

Delivery  Forecast  Timeliness  DCMC 

Negotiation  Cycle  Time  DCMC 

Percent  of  Schedules  On-Time  DCMC 

Response  Time  to  Customer  Priority  Surveillance  DCMC 

System  (CPSS)  Requests 

Schedule  Slippage's  on  Major  Programs  DCMC 

Product  Availability  DLA 

RFP  Review  Participation  DLA 

Customer  Satisfaction  DLA 

On-Time  Contractor  Delivery  DLA 

Logistics  Response  Time  DLA 

Oversight  of  Management  of  Prime  Contractor  DOE 
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Compliance 

Percent  of  Acquisitions  Awarded  to  SDB  and  Women-  DOE 
owned  Business 

Percent  of  Non-competitive  Acquisitions  Documented  DOE 
with  Justification 

Percent  of  Total  Subcontract  Dollars  Awarded  on  DOE 

Competitive  Basis 

Identify  Processes  Used  to  Reduce  Cycle  Time  and  DOE 

Operating  Costs 


Responsiveness  of  DM  Organizations  Pertaining  to  DOE  DOE 
Directions 

Percent  of  Acquisitions  Processed  within  Established  DOE 
Lead  Times 


Competition  NASA 
Extent  of  Commercial  Items  Procurement  NASA 
Performance-Based  Contracting  (Extent  of  use)  NASA 
Support  to  Socio-economic  Programs  NASA 
Total  number  of  1100  series  personnel  NASA 
(engaged  in  combination  of  award  and  administration) 

Ratio  of  Personnel  to  Actions  (1:XX)  NASA 
Electronic  Commerce  (EC)  NASA 
Cost-to-Manage  NASA 
Financial  Actions  (New  Awards  and  Modifications)  NASA 
Unliquidated  Obligations  (ULOs)  NASA 
New  Contracts,  Competitive/Non-Competitive  NASA 
Modifications,  Administrative/Incremental  NASA 
Funding/Other  "Dollar"  Modifications 

Actions  under  $100K  NASA 
(Purchase  Orders,  Simplified  Acquisitions) 

New  Financial  Assistance  Instruments  NASA 
(Grants,  Cooperative  Agreements) 

Modifications  to  Financial  Assistance  Instruments  NASA 
Other  Intra-governmental ,  Space  Act  agreements,  NASA 
Orders  under  FSS 

Close-out  -  Number  of  awards  pending  NASA 
Close-out,  volume  NASA 
Consolidated  Contracting  Initiative  (CCI)  NASA 
Contract  actions  (New  awards  and  modifications)  NASA 
Other  Actions  NASA 
Procurement  Workload  NASA 
Purchase  Orders/Simplified  Acquisitions  NASA 
Customer  Satisfaction  NASA 
GAO/Agency  Protests  NASA 
Procurement  Training  NASA 
Award  Fee,  Timeliness  of  Payment  NASA 
Close-out  (Timeliness)  NASA 
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DCAA  Audits  (Timeliness  of  audit  resolution) 

Lead  Time 

Unde f ini ti zed  Contract  Actions  (UCAs) 

Actual  Awards  Compared  to  Execution  Plan  Goals 
Counts  of  WIP  By  Contracting  Departments,  Divisions 
and  Buyers 

List  of  Top  ACOs  with  Outstanding  Terminations 
List  of  Top  ACOs  with  Outstanding  UCAs 
List  of  Top  Contractors  with  Outstanding 
Terminations 

List  of  Top  Contractors  with  Outstanding 
Undef initized  Contractual  Actions  (UCAs) 

Detailed  Automated  Work  In  Process  Database 
Detailed  Manual  Work  In  Process  Database 
Dollar  Values  and  Counts  of  Deobligations 
Dollar  Values  and  Counts  of  Outstanding  Contracts 
and  Outstanding  Contracts  w/Back  Orders 
Dollar  Values  and  Counts  of  Outstanding  UCAs  By 
Weapon  System  Teams 

Dollar  Values  and  Counts  of  Overage  Procurements 
Dollar  Values  of  Receipts,  Pre  Award  DTs, 
Obligations,  Credits  and  Work  In  Process 
Counts  of  Work  In  Process  (WIP)  By  Procurement  Types 
and  By  Various  Age 

Dollar  Values  and  Counts  of  Completed  Actions 
(Awards) 

Dollar  Values  and  Counts  of  New  Work  In  Process 
Receipts 

Dollar  Values  and  Counts  of  Post  Award  DTs 
Obligations  summaries  by  Procurement  Type 
(Repair  or  Spares)  and  Budget  Plan  (BP) 

Averages  of  Procurement  Administrative  Lead  Time 
(PALT) 

Counts  of  Overage  WIP  By  Contracting  Departments, 
Divisions  and  Buyers 

Counts  of  Overage  WIP  By  Weapon  System  Teams 
Detailed  Delinquent  Contracts  with  Back  Orders 
Detailed  Outstanding  UCAs  Database 

HQ  Contracting  Actions  vs.  NAVSEA  02  Staffing 
NAVSEA  02  Manning 
Paperless  Acquisition 

Top  Five  Contractors  by  Dollar  (Post  COFD) 

Top  Five  Contractors  by  Dollar  (Pre  COFD) 

HQ  and  Field  Competitive  Awards 
(Percent  of  Dollars  Obligated) 


NASA 

NASA 

NASA 

NAVICP 

NAVICP 

NAVICP 

NAVICP 

NAVICP 

NAVICP 

NAVICP 

NAVICP 

NAVICP 

NAVICP 

NAVICP 

NAVICP 

NAVICP 

NAVICP 

NAVICP 

NAVICP 

NAVICP 

NAVICP 

NAVICP 

NAVICP 

NAVICP 

NAVICP 

NAVICP 

NAVSEA 

NAVSEA 

NAVSEA 

NAVSEA 

NAVSEA 

NAVSEA 
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Purchase  Card  Sales  &  Delinquencies 
Purchasing  Costs  in  Cents  per  Dollar  Obligated 
Field  Contract  Awards  (Dollars  &  Actions) 

HQ  Contract  Awards  (Dollars  &  Actions) 

Accessibility  of  Senior  People 

Attitude,  Responsiveness,  Accountability 

Comparison  to  similar  Government  Activities 

Keeping  you  informed 

Quality  of  Work 

Technical  Capabilities 

Value  Received 

Contract  Cases  in  Litigation 

(ASBCA,  COFC,  Appellate  Courts,  District  Courts) 
Customer  Survey  Results 
GAO  Protest  (HQ  &  Field) 

REA  &  Claims  at  HQ/Field  (Pre-COFD,  Post-COFD) 
Shipbuilder's  Disputes  in  Contract  Litigation  (FY) 
Timeliness 

Pre  &  Post  COFD  (Dollars)  and  Pre  COFD  Aged  Analysis 

REA  &  Claims  at  HQ/Field  >3  years  old 

REA  &  Claims  at  HQ/Field  1-2  years  old 

REA  &  Claims  at  HQ/Field  2-3  years  old 

UCA  (Total  &  >180  Days) 

Best  and  Final  Offers  ( BAFO )  Report  of  Multiple 

issuance 

Competition 

Competition  Goals 

Competition  Initiatives 

Purchase  Card  -  Express  Software  Requests 

Small  Business  Goals 

Contracting  Authority 

Electronic  Data  Interchange 

Paperless  Acquisition 

Staffing 

Workload  And  Staffing 

Cost  To  Obligate  $1 

Other  Than  Purchase  Card  By  $$ 

Purchase  Card  -  Delinquencies  >  90  Days  Credit 
Balance  Invoices 

Purchase  Card  -  Delinquencies  >  90  Days  Debit 
Balance  Invoices 

Purchase  Card  -  Delinquencies  >  90  Days  Zero 
Balances  (Interest  Only)  Invoices 
Purchase  Card  -  Delinquency  By  Claimant 
Purchase  Card  -  Delinquency  Tiger  Team 
Contract  Actions 


NAVSEA 

NAVSEA 

NAVSEA 

NAVSEA 

NAVSEA 

NAVSEA 

NAVSEA 

NAVSEA 

NAVSEA 

NAVSEA 

NAVSEA 

NAVSEA 

NAVSEA 

NAVSEA 

NAVSEA 

NAVSEA 

NAVSEA 

NAVSEA 

NAVSEA 

NAVSEA 

NAVSEA 

NAVSEA 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 
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Contract  Close-out  (volume) 

Extraordinary  Contractual  Actions 
Government  Property 
Other  Than  Purchase  Card 
Purchase  Card  -  Growth 
(DoD  Actions/Card  Holders/Sales) 

Purchase  Card  -  Growth 

(Navy  Actions /Card  Holders /Sales) 

Purchase  Card  -Usage  Afloat 

Simplified  Acquisition  Procedures  -  Actions 
Claims 

Contract  Audit  Follow  Up 
Customer  Sat  Ratings 
Customer  Service 
Customer  Service  Standards 
Procurement  Management  Review (PMR) 

Red/Yellow/Green 

Contract  Close-out  (Timeliness) 

Cycle  Time  Reduction 
PALT  -  5  Year  History 
PALT  -  Monthly 
PALT  -  Piers ide 

PALT  -  Simplified  Acquisition  $25K  -  $100K 
Undef initized  Contractual  Actions  (UCAs) 

Number  of  Predetermined  Pricing  Rates  -  Grants 

w/Universities 

Repr o  curements 

EDI  Vouchers  Processed 

Manual  Vouchers  Processed 

Workload  ACO  FTE  Vs  Number  of 

Contracts /CA/OTA/Grants 

Workload  FTE  Vs  Awards 

Contractor  Performance  Assessment  Reporting  System 
(Dollar  Value  of  Contracts) 

Purchase  Card  Obligations 
Contract  Awards 

Contractor  Performance  Assessment  Reporting  System 
(Number  of  Contracts) 

Contractor  Performance  Assessment  Reporting  System 
(Number  of  Contracts  Completed) 

DAU  Utilization  Rate 
Grants  Awards 

Number  of  Actions  -  Contracts 

Number  of  Actions  -  Cooperative  Agreements /OTA 
Number  of  Actions  -  Grants 
Purchase  Card  -Usage 


NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

ONR 

ONR 

ONR 

ONR- 

ONR 

ONR 

ONR 

ONR 

ONR 

ONR 

ONR 

ONR 

ONR 

ONR 

ONR 

ONR 

ONR 
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1102  Certif ication/DAWIA  Training  ONR 
Claims  ONR 
Protests  ONR 
PALT  -  Contracts  ONR 
PALT  -  Grants  ONR 
UCA  ONR 


%  Contracts  Closed  Out  Automatically 
Dollars  Awarded  Electronically 
PO  Issued  Electronically 
Solicitation  Posted  on  WWW 

%  Awards /Modifications  Transactions  Automated 
%  Funding  Documents  Automated 
%  PR  Automated 
%  Solicitations  Automated 

Awards  and  Modifications  Transactions  Electronic 
Awards  and  Modifications  sent  to  electronically: 
Files,  Originator,  ACO,  DFAS  Pay,  DFAS  Accounting, 
Contractor 

Funding  Documents  Transmitted 
Funding  Documents  Transmitted  Electronically 
No.  Contracts  Closed  Out  Electronically 
No.  Receipt  transactions  Electronic 
PR  Transmitted 

PR  Transmitted  Electronically 
Solicitations  Issued  Electronically/Orally 
Awards  and  Modifications  Transactions 
No.  Contracts  Closed  Out 
No.  Receipt  transactions 
Solicitations  Issued 


SPAWAR 

SPAWAR 

SPAWAR 

SPAWAR 

SPAWAR 

SPAWAR 

SPAWAR 

SPAWAR 

SPAWAR 

SPAWAR 


SPAWAR 

SPAWAR 

SPAWAR 

SPAWAR 

SPAWAR 

SPAWAR 

SPAWAR 

SPAWAR 

SPAWAR 

SPAWAR 

SPAWAR 
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APPENDIX  B.  CONTRACT  MANAGEMENT  METRICS  BY  MEASUREMENT 
CLASSIFICATION  CATEGORY 


Metric 

Category 

M/C  Request  for  Waivers /Request  For  Deviations  per 
1,000  Contracts 

DCMC 

EFFECT 

Major  Software  Recommendations  Adopted 

DCMC 

EFFECT 

Product  Availability 

DLA 

EFFECT 

Oversight  of  Management  of  Prime  Contractor 

Compliance 

DOE 

EFFECT 

Percent  of  Acquisitions  Awarded  to  SDB  and  Women- 
owned  Business 

DOE 

EFFECT 

Percent  of  Non-competitive  Acquisitions  Documented 
with  Justification 

DOE 

EFFECT 

Percent  of  Total  Subcontract  Dollars  Awarded  on 
Competitive  Basis 

DOE 

EFFECT 

Competition 

NASA 

EFFECT 

Extent  of  Commercial  Items  Procurement 

NASA 

EFFECT 

Performance-Based  Contracting  (Extent  of  use) 

NASA 

EFFECT 

Support  to  Socio-economic  Programs 

NASA 

EFFECT 

Actual  Awards  Compared  to  Execution  Plan  Goals 

NAVI CP 

EFFECT 

Counts  of  WIP  By  Contracting  Departments,  Divisions 
and  Buyers 

NAVICP 

EFFECT 

List  of  Top  ACOs  with  Outstanding  Terminations 

NAVICP 

EFFECT 

List  of  Top  ACOs  with  Outstanding  UCAs 

NAVICP 

EFFECT 

List  of  Top  Contractors  with  Outstanding  Terminations 

NAVICP 

EFFECT 

List  of  Top  Contractors  with  Outstanding 

Undef initized  Contractual  Actions  (UCAs) 

NAVICP 

EFFECT 

Best  and  Final  Offers  (BAFO)  -  Report  of  Multiple 
issuance 

NAVSUP 

EFFECT 

Competition 

NAVSUP 

EFFECT 

Competition  Goals 

NAVSUP 

EFFECT 

Competition  Initiatives 

NAVSUP 

EFFECT 

Purchase  Card  -  Express  Software  Requests 

NAVSUP 

EFFECT 

Small  Business  Goals 

NAVSUP 

EFFECT 

Number  of  Predetermined  Pricing  Rates  -  Grants 
w/Universities 

ONR 

EFFECT 

Reprocurements 

ONR 

EFFECT 

%  Contracts  Closed  Out  Automatically 

SPAWAR 

EFFECT 

Identify  Processes  Used  to  Reduce  Cycle  Time  and 
Operating  Costs 

DOE 

EFFICIENCY 

Active  Suppliers  per  Professional  Purchasing  Employee  CAPS 

EFFICIENCY 

Active  Suppliers  per  Purchasing  Employee 

CAPS 

EFFICIENCY 

Percent  of  Purchase  Transactions  Processed  through 
Electronic  Commerce 

CAPS 

EFFICIENCY 

Percent  of  Purchase  Transactions  Processed  via 
Procurement  Card 

CAPS 

EFFICIENCY 

Percent  of  Services  Purchases  handled  by  the 
Purchasing  Department 

CAPS 

EFFICIENCY 

Percent  of  Total  Purchases  handled  by  the  Purchasing 
Department 

CAPS 

EFFICIENCY 

Purchase  Dollars  per  Professional  Purchasing  Employee  CAPS 

EFFICIENCY 

Purchase  Dollars  per  Purchasing  Employee 

CAPS 

EFFICIENCY 

Purchasing  Employees  as  a  Percent  of  Company  Employee  CAPS 

EFFICIENCY 
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Corrective  Action  Request  Cost  Avoidance 
Quantity  of  Overage  Contracts  for  Closure 
Unliquidated  Amount  of  Prime  Contracts  On-Hand 
RFP  Review  Participation 
Electronic  Commerce  (EC) 

Ratio  of  Personnel  to  Actions  (1:XX) 

Total  number  of  1100  series  personnel  (engaged  in 
combination  of  award  and  administration  ) 

Detailed  Automated  Work  In  Process  Database 

Detailed  Manual  Work  In  Process  Database 

HQ  Contracting  Actions  vs.  NAVSEA  02  Staffing 

NAVSEA  02  Manning 

Paperless  Acquisition 

Contracting  Authority 

Electronic  Data  Interchange 

Paperless  Acquisition 

Staffing 

Workload  And  Staffing 
EDI  Vouchers  Processed 
Manual  Vouchers  Processed 

Workload  ACO  FTE  Vs  Number  of  Contracts /CA/ OTA/ Grants 
Workload  FTE  Vs  Awards 

%  Awards /Modifications  Transactions  Automated 
%  Funding  Documents  Automated 
%  PR  Automated 
%  Solicitations  Automated 

Awards  and  Modifications  Transactions  Electronic 
Awards  and  Modifications  sent  to  electronically: 
Files,  Originator,  ACO,  DFAS  Pay,  DFAS  Accounting, 
Contractor 


DCMC 

DCMC 

DCMC 

DLA 

NASA 

NASA 

NASA 

NAVI CP 

NAVI CP 

NAVSEA 

NAVSEA 

NAVSEA 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

NAVSUP 

ONR 

ONR 

ONR 

ONR 

SPAWAR 

SPAWAR 

SPAWAR 

SPAWAR 

SPAWAR 

SPAWAR 


Dollars  Awarded  Electronically 

Funding  Documents  Transmitted 

Funding  Documents  Transmitted  Electronically 

No.  Contracts  Closed  Out  Electronically 

No.  Receipt  transactions  Electronic 

PO  Issued  Electronically 

PR  Transmitted 

PR  Transmitted  Electronically 
Solicitation  Posted  on  WWW 

Solicitations  Issued  Electronically/Orally 


SPAWAR 

SPAWAR 

SPAWAR 

SPAWAR 

SPAWAR 

SPAWAR 

SPAWAR 

SPAWAR 

SPAWAR 

SPAWAR 


EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 


EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 

EFFICIENCY 


Funding  Documents  Received  AFRL 

Reduce  Total  Ownership  Costs  (Development,  ASN 

Acquisition,  O&S) 

Return  on  Investment  (ROI)  [TOC/ INVESTMENT]  ASN 

Cost  To  Spend  A  Dollar  CAPS 

Percent  of  Purchase  Dollars  Spent  with  Minority- owned  CAPS 
Suppliers 

Percent  of  Purchase  Dollars  Spent  with  Women-owned  CAPS 
Suppliers 

Purchase  Dollars  as  a  Percent  of  Sales  Dollars  CAPS 

Purchase  Dollars  Spent  per  Active  Supplier  CAPS 

Purchase  Operating  Expense  Dollars  as  a  Percent  of  CAPS 
Sales  Dollars 

Purchasing  Operating  Expense  Dollars  per  Active  CAPS 

Supplier 


FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 
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Amount  of  Loss,  Damage  and  Destruction  (LDD)  DCMC 

Contracting  Officer  Price  Negotiations  Savings  and  DCMC 
Avoidance 

Cost  Overruns  on  Major  Programs  DCMC 

Litigation  Cost  Savings  and  Avoidance  DCMC 

Percent  of  Excess  Property  Reutilized  and  Sales  DCMC 

Proceeds 

Process  Improvement  Cost  Savings  and  Avoidance  DCMC 

Quantity  of  Price  Negotiations  (Used  to  monitor  cost  DCMC 
of  negotiations) 

Cost-to-Manage  NASA 

Financial  Actions  (New  Awards  and  Modifications)  NASA 

Unliquidated  Obligations  (ULOs)  NASA 

Dollar  Values  and  Counts  of  Deobligations  NAVICP 

Dollar  Values  and  Counts  of  Outstanding  Contracts  and  NAVICP 
Outstanding  Contracts  w/Back  Orders 

Dollar  Values  and  Counts  of  Outstanding  UCAs  By  NAVICP 

Weapon  System  Teams 

Dollar  Values  and  Counts  of  Overage  Procurements  NAVICP 

Dollar  Values  of  Receipts,  Pre  Award  DTs ,  NAVICP 

Obligations,  Credits  and  Work  In  Process 

HQ  and  Field  Competitive  Awards  (Percent  of  Dollars  NAVSEA 
Obligated) 

Purchase  Card  Sales  &  Delinquencies  NAVSEA 

Purchasing  Costs  in  Cents  per  Dollar  Obligated  NAVSEA 

Top  Five  Contractors  by  Dollar  (Post  COFD)  NAVSEA 

Top  Five  Contractors  by  Dollar  (Pre  COFD)  NAVSEA 

Cost  To  Obligate  $1  NAVSUP 

Other  Than  Purchase  Card  By  $$  NAVSUP 

Purchase  Card  -  Delinquencies  >  90  Days  Credit  NAVSUP 


Balance  Invoices 

Purchase  Card  -  Delinquencies  >  90  Days  Debit  Balance  NAVSUP 
Invoices 

Purchase  Card  -  Delinquencies  >  9  0  Days  Zero  Balances  NAVSUP 


(Interest  Only)  Invoices 

Purchase  Card  -  Delinquency  By  Claimant  NAVSUP 

Purchase  Card  -  Delinquency  Tiger  Team  NAVSUP 

Contractor  Performance  Assessment  Reporting  System  -  ONR 
Dollar  Value  of  Contracts 

Purchase  Card  -Obligations  ONR 


FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 

FINANCIAL 


Awards  &  Modifications  Issued  AFRL 
Purchase  Requests  Transmitted  AFRL 
Solicitations  Issued  AFRL 
Change  in  Number  of  Active  Suppliers  During  the  CAPS 
Reporting  Period 

Percent  of  Active  Suppliers  Accounting  for  90  Percent  CAPS 
of  Purchase  Dollars 

Sales  Dollar  per  Purchasing  Employee  CAPS 
Contractors  Assigned  Prime  Contracts  DCMC 
Obligated  Amount  of  Prime  Contracts  On-Hand  DCMC 
Prime  Contracts  On-Hand  DCMC 
Actions  under  $100K  (Purchase  Orders,  Simplified  NASA 
Acquisitions ) 

Close-out  -  Number  of  awards  pending  NASA 
Close-out,  volume  NASA 
Consolidated  Contracting  Initiative  (CCI)  NASA 


PRODUCT 

PRODUCT 

PRODUCT 

PRODUCT 

PRODUCT 

PRODUCT 

PRODUCT 

PRODUCT 

PRODUCT 

PRODUCT 

PRODUCT 

PRODUCT 

PRODUCT 
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Contract  actions  (New  awards  and  modifications) 
Modifications  to  Financial  Assistance  Instruments 
Modifications ,  Administrative/ Incremental 
Funding/Other  "Dollar"  Modifications 
New  Contracts,  Competitive /Non-Competitive 
New  Financial  Assistance  Instruments  (Grants, 
Cooperative  Agreements) 

Other  ( Intra-Governmental ,  Space  Act  agreements. 
Orders  under  FSS) 

Other  Actions 
Procurement  Workload 

Purchase  Orders/ Simplified  Acquisitions 

Counts  of  Work  In  Process  (WIP)  By  Procurement  Types 

and  By  Various  Age 

Dollar  Values  and  Counts  of  Completed  Actions 
(Awards) 

Dollar  Values  and  Counts  of  New  Work  In  Process 
Receipts 

Dollar  Values  and  Counts  of  Post  Award  DTs 
Obligations  summaries  by  Procurement  Type  (Repair  or 
Spares)  and  Budget  Plan  (BP) 

Field  Contract  Awards  (Dollars  &  Actions) 

HQ  Contract  Awards  (Dollars  &  Actions) 

Contract  Actions 

Contract  Close-out  (volume) 

Extraordinary  Contractual  Actions 
Government  Property 
Other  Than  Purchase  Card 

Purchase  Card  -  Growth  (DoD  Actions /Card 
Holders /Sales) 

Purchase  Card  -  Growth  (Navy  Actions /Card 
Holders /Sales) 

Purchase  Card  -Usage  Afloat 

Simplified  Acquisition  Procedures  -  Actions 
Contract  Awards 

Contractor  Performance  Assessment  Reporting  System  - 
Number  of  Contracts 

Contractor  Performance  Assessment  Reporting  System  - 
Number  of  Contracts  Completed 
DAU  Utilization  Rate 
Grants  Awards 

Number  of  Actions  -  Contracts 

Number  of  Actions  -  Cooperative  Agreements /OTA 
Number  of  Actions  -  Grants 
Purchase  Card  -Usage 

Awards  and  Modifications  Transactions 
No.  Contracts  Closed  Out 
No.  Receipt  transactions 
Solicitations  Issued 


NASA  PRODUCT 
NASA  PRODUCT 
NASA  PRODUCT 

NASA  PRODUCT 
NASA  PRODUCT 

NASA  PRODUCT 

NASA  PRODUCT 
NASA  PRODUCT 
NASA  PRODUCT 
NAVICP  PRODUCT 

NAVI CP  PRODUCT 

NAVICP  PRODUCT 

NAVICP  PRODUCT 
NAVICP  PRODUCT 

NAVSEA  PRODUCT 
NAVSEA  PRODUCT 
NAVSUP  PRODUCT 
NAVSUP  PRODUCT 
NAVSUP  PRODUCT 
NAVSUP  PRODUCT 
NAVSUP  PRODUCT 
NAVSUP  PRODUCT 

NAVSUP  PRODUCT 

NAVSUP  PRODUCT 
NAVSUP  PRODUCT 
ONR  PRODUCT 
ONR  PRODUCT 

ONR  PRODUCT 

ONR  PRODUCT 
ONR  PRODUCT 
ONR  PRODUCT 
ONR  PRODUCT 
ONR  PRODUCT 
ONR  PRODUCT 
SPAWAR  PRODUCT 
SPAWAR  PRODUCT 
SPAWAR  PRODUCT 
SPAWAR  PRODUCT 


Average  Annual  Training  Hours  per  Professional 
Purchasing  Employee 

CAPS 

QUALITY 

Completeness  of  the  Contractor  Alert  List  (CAL) 

DCMC 

QUALITY 

Customer  Satisfaction  Rating 

DCMC 

QUALITY 

Grievances 

DCMC 

QUALITY 

Customer  Satisfaction 

DLA 

QUALITY 

On-Time  Contractor  Delivery 

DLA 

QUALITY 
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Responsiveness  of  DM  Organizations  Pertaining  to  DOE 
Directions 

DOE 

QUALITY 

Customer  Satisfaction 

NASA 

QUALITY 

GAO/Agency  Protests 

NASA 

QUALITY 

Procurement  Training 

NASA 

QUALITY 

Accessibility  of  Senior  People 

NAVSEA 

QUALITY 

Attitude,  Respons i venes  s ,  Ac  c ount abi 1 i ty 

NAVSEA 

QUALITY 

Comparison  to  similar  Government  Activities 

NAVSEA 

QUALITY 

Contract  Cases  in  Litigation  (ASBCA,  COFC,  Appellate 
Courts,  District  Courts) 

NAVSEA 

QUALITY 

Customer  Survey  Results 

NAVSEA 

QUALITY 

GAO  Protest  (HQ  &  Field) 

NAVSEA 

QUALITY 

Keeping  you  informed 

NAVSEA 

QUALITY 

Quality  of  Work 

NAVSEA 

QUALITY 

REA  Sc  Claims  at  HQ/Field  (Pre-COFD,  Post-COFD) 

NAVSEA 

QUALITY 

Shipbuilder's  Disputes  in  Contract  Litigation  (FY) 

NAVSEA 

QUALITY 

Technical  Capabilities 

NAVSEA 

QUALITY 

Value  Received 

NAVSEA 

QUALITY 

Claims 

NAVSUP 

QUALITY 

Contract  Audit  Follow  Up 

NAVSUP 

QUALITY 

Customer  Sat  Ratings 

NAVSUP 

QUALITY 

Customer  Service 

NAVSUP 

QUALITY 

Customer  Service  Standards 

NAVSUP 

QUALITY 

Procurement  Management  Review (PMR) 

NAVSUP 

QUALITY 

Red/ Yellow/Green 

NAVSUP 

QUALITY 

1102  Certification/DAWIA  Training 

ONR 

QUALITY 

Claims 

ONR 

QUALITY 

Protests 

ONR 

QUALITY 

Purchase  Requests  Transmitted  to  DFAS 

AFRL 

TIME 

Purchase  Requests  Transmitted  to  Files /Archives 

AFRL 

TIME 

Purchase  Requests  Transmitted  to  Originator 

AFRL 

TIME 

Solicitations  Transmitted  to  ACO 

AFRL 

TIME 

Solicitations  Transmitted  to  Contractors 

AFRL 

TIME 

Reduce  Major  Product  Cycle  Time  -  Time  to  Develop  and  ASN 
Field  System 

TIME 

Reduce  Time  to  Develop  and  Insert  New  Technologies 

ASN 

TIME 

Shorten  Product  Improvement  Cycle  -  Time  to  Develop 
and  Field  Major  Modifications 

ASN 

TIME 

Purchase  Order  Cycle  Time  (days) 

CAPS 

TIME 

Contract  Closeout  Cycle  Time 

DCMC 

TIME 

Delivery  Forecast  Timeliness 

DCMC 

TIME 

Negotiation  Cycle  Time 

DCMC 

TIME 

Percent  of  Schedules  On-Time 

DCMC 

TIME 

Response  Time  to  Customer  Priority  Surveillance 

System  (CPSS)  Requests 

DCMC 

TIME 

Schedule  Slippage's  on  Major  Programs 

DCMC 

TIME 

Logistics  Resppnse  Time 

DLA 

TIME 

Percent  of  Acquisitions  Processed  within  Established 
Lead  Times 

DOE 

TIME 

Award  Fee,  Timeliness  of  Payment 

NASA 

TIME 

Close-out  (Timeliness) 

NASA 

TIME 

DCAA  Audits  (Timeliness  of  audit  resolution) 

NASA 

TIME 

Lead  Time 

NASA 

TIME 

Undef initized  Contract  Actions  (UCAs) 

NASA 

TIME 

Averages  of  Procurement  Administrative  Lead  Time 
(PALT) 

NAVI CP 

TIME 
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NAVI CP  TIME 


Counts  of  Overage  WIP  By  Contracting  Departments, 
Divisions  and  Buyers 

Counts  of  Overage  WIP  By  Weapon  System  Teams 
Detailed  Delinquent  Contracts  with  Back  Orders 
Detailed  Outstanding  UCAs  Database 

Pre  &  Post  COFD  (Dollars)  and  Pre  COFD  Aged  Analysis 

REA  &  Claims  at  HQ/Field  >3  years  old 

REA  &  Claims  at  HQ/Field  1-2  years  old 

REA  &  Claims  at  HQ/Field  2-3  years  old 

Timeliness 

UCA  (Total  &  >180  Days) 

Contract  Close-out  (  timeliness) 

Cycle  Time  Reduction 
PALT  -  5  Year  History 
PALT  -  Monthly 
PALT  -  Piers ide 

PALT  -  Simplified  Acquisition  $25K  -  $100K 
Undef initized  Contractual  Actions  (UCAs) 

PALT  -  Contracts 
PALT  -  Grants 
UCA 


NAVI CP  TIME 
NAVI CP  TIME 
NAVI CP  TIME 
NAVSEA  TIME 
NAVSEA  TIME 
NAVSEA  TIME 
NAVSEA  TIME 
NAVSEA  TIME 
NAVSEA  TIME 
NAVSUP  TIME 
NAVSUP  TIME 
NAVSUP  TIME 
NAVSUP  TIME 
NAVSUP  TIME 
NAVSUP  TIME 
NAVSUP  TIME 
ONR  TIME 
ONR  TIME 
ONR  TIME 
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APPENDIX  C 


TWENTY-FIVE  STANDARD  CONTRACT  METRICS 
SUBJECTIVELY  DETERMINED  TO  BE  MOST  RELEVANT  TO 
MEASURING  SECTION  845  OTHER  TRANSACTIONS 


metric  category  activity 


Awards  and  Modifications  Issued 

EFFECT 

AFRL 

Competition  Goals 

EFFECT 

NAVSUP 

Active  Suppliers  Per  Purchasing  Employee 

EFFICIENCY 

CAPS 

Workload  And  Staffing 

EFFICIENCY 

NAVSUP 

Reduce  Total  Ownership  Costs  (TOC) : 

FINANCIAL 

ASN 

Development,  Acquisition,  O&S 

Return-On- Investment  (ROI) 

FINANCIAL 

ASN 

[TOC /INVESTMENT] 

Cost-To-Spend  A  Dollar 

FINANCIAL 

CAPS 

Cost-To-Manage 

FINANCIAL 

NASA 

Solicitations  Issued 

PRODUCT 

AFRL 

Contractors  Assigned  Prime  Contracts 

PRODUCT 

DCMC 

New  Contracts,  Competitive/Non- 

PRODUCT 

NASA 

Competitive 

New  Financial  Assistance  Instruments 

PRODUCT 

NASA 

(Grants,  Cooperative  Agreements) 

Contract  Awards 

PRODUCT 

ONR 

Customer  Satisfaction 

QUALITY 

NAVSEA 

Value  Received 

QUALITY 

NAVSEA 

Customer  Service 

QUALITY 

NAVSUP 

Reduce  Time  to  Develop  and  Insert  New 

TIME 

ASN 

Technologies 

Reduce  Major  Product  Cycle  Time  (Time  to 

TIME 

ASN 

Develop  &  Field  Major  System) 

Shorten  Product  Improvement  Cycle  (Time 

TIME 

ASN 

to  Develop  &  Field  Major  Modifications) 

Purchase  Order  Cycle  Time  (Days) 

TIME 

CAPS 

Negotiation  Cycle  Time 

TIME 

DCMC 

Lead  Time 

TIME 

NASA 

Timeliness 

TIME 

NAVSEA 

Cycle  Time  Reduction 

TIME 

NAVSUP 

Procurement  Administration  Lead  Time 

TIME 

ONR 

(Contracts) 
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APPENDIX  D.  TERMS  AND  DEFINITIONS 


1.  Other  Transactions  (OT)  for  Research:  OTs  are  cost 
shared  agreements  that  support  basic,  applied  and 
advanced  research.  They  are  funding  transactions  that 
are  not  contracts,  grants,  or  cooperative  agreements. 
This  type  of  OT  authority  has  generally  been  used  to 
enter  into  dual-use  research  projects.  [Ref.  29] 

2 .  Section  845  Other  Transactions :  Also  referred  to  as 
"Other  Transactions  for  Prototypes",  are  directly 
relevant  to  weapons  or  weapons  systems  prototypes 
proposed  to  be  acquired  or  developed  by  Department  of 
Defense  (DOD) .  [Ref.  29] 

3.  Contracts:  Are  a  mutually  binding  legal  relationship 
obligating  the  seller  to  furnish  the  supplies  or 
services  (including  construction)  and  the  buyer  to  pay 
for  them.  It  includes  all  types  of  commitments  that 
obligate  the  Government  to  an  expenditure  of 
appropriated  funds  and  that,  except  as  otherwise 
authorized,  are  in  writing.  Contracts  can  be  cost 
sharing  instruments,  but  are  normally  solely  funded  by 
the  government.  Contracts  do  not  include  grants  and 
cooperative  agreements  covered  by  31  U.S.C.  6301. 

[Ref.  DOD  7000. 14-R] 

4.  Grant:  Grants  are  assistance  awards  for  which  no 
substantial  involvement  is  anticipated  between  the 
Department  of  Defense  and  the  recipient  during 
performance  of  the  contemplated  activity.  [Ref.  FAR 
Part  2] 

5.  Cooperative  Agreement:  Unlike  grants,  cooperative 
agreements  require  substantial  involvement  on  the  part 
of  DOD.  Recipients  of  DOD  grants  and  cooperative 
agreements  include  individuals,  nonprofit 
organizations,  local  and  state  governments,  U.S. 
territories,  and  federally  recognized  tribal 
governments.  [Ref.  FAR  Part  2] 


163 


What  grants  and  cooperative  agreements  are  not : 

Because  they  are  assistance  instruments,  rather  than 
acquisition  instruments,  they  are  subject  to  a 
different  set  of  rules  and  laws  than  the  procurement 
laws  and  FAR.  They  do  not  include;  (1)  technical 
assistance  programs  that  provide  services  instead  of 
money;  (2)  contracts  that  are  entered  into  and 
administered  under  procurement  laws  and  regulations; 
and  (3)  agreements  under  which  only  direct  cash 
assistance  to  individuals,  a  subsidy,  a  loan,  a  loan 
guarantee,  or  insurance  is  provided.  [Ref.  FAR  Part  2] 
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